eBPF with Nix: laptop to testbed

Yifei Sun

Inria, ENS de Lyon, Université Grenoble Alpes

Background

I started a project

• Multicast caching system for networked FS over XDP
• Its running late
• So here I am…

Background

Problem

• Environment setup and collboration

  • Headers, compiler, editor…
  • KConfig, QEMU, … (what if multiple machine is needed?)

• Development, deployment and benchmark

  • Cluster boot, data collection, …

• Peer review

  • Ease of result reproduction

What worked for me

NixOS VM tests

• Basically Nix + Python + QEMU
• Multi-machines, different kernels, networking
• Binary cache, and benefits from using Nix

NixOS Compose

• Deployment tool
• Substitute with your own stuff

Userspace tooling

Pull packages from pinned nixpkgs

devShells.x86_64-linux.default = pkgs.mkShell {
  inputsFrom = [ <locally defined derivations> ];
  packages = with pkgs.llvmPackages; [ clang-unwrapped libllvm ];
};

• Compilers
• Libraries
• …

Get a kernel

One machine

pkgs.testers.runNixOSTest {
  name = "one-machine-test";

  nodes.machine1 = {
    imports = [ nixosModules.kernel ];
    services.scx.enable = true;
  };

  testScript = ''
    machine1.wait_for_unit("default.target")
    machine1.succeed("")
    machine1.fail("")
  '';
}

More machines?

pkgs.testers.runNixOSTest {
  name = "lots-of-machine-test";

  nodes.machine1.imports = [ nixosModules.grafana ];
  nodes.machine2.imports = with nixosModules; [
    kernel exporter ebpf benchmark
  ];

  testScript = ''
    start_all()
    machine1...
    machine2...
  '';
}

What’s in there?

Mock syscall

Say we want to troll ourselves:

SEC("ksyscall/statx")
int BPF_KSYSCALL(fsd_statx_entry, ... statx(2) args) {
  // generate a map entry to collect start ts
  // check path, if not match return
  // else override with a static statx content
  struct statx stx = { ... };
  bpf_probe_write_user(statxbuf, &stx, sizeof(stx));
  return bpf_override_return(ctx, 0);
}

And count how many times we can footgun ourselves

With a counter and a histogram

Declarative userspace program

• Auto-load the the program (feat. ebpf_exporter)
• Collect the metrics and plot them (feat. Prometheus & Grafana)
• Local testing (feat. NixOS VM test)
• Deployment (feat. NixOS-Compose)

Local testing

For simplicity

• We will be using a readily available userspace tool

  • Loading the program
  • Read the map and re-expose the content over Prometheus

Complication is fast

• Build once and its immutable
• Push cache to server (or have a CI server build it)
• SBOM

Debugging is easy

• SSH backdoor enable with a knob

Demo

• Build interactive driver closure

  nom build .#checks.x86_64-linux.default.driverInteractive

• Start the driver

$ ./result/bin/nixos-test-driver
start vlan
running vlan (pid 3859017; ctl /run/user/1000/vde1.ctl)
SSH backdoor enabled, the machines can be accessed like this:
    collector:  ssh -o User=root vsock/3
    exporter:   ssh -o User=root vsock/4

Straight to prod

Bit-perfect reproducibility (*: for some store paths)

Everything is in closure

• Deployment harness is easy to write

Demo

• Build deployment closure (instrumented with NixOS test)

  nxc build

• Schedule couple machines and deploy the closure to cluster

Effort