Table of Contents

03. [30p] Packets, where are you?

Earlier in Ex. 1, we mentioned that eBPF is used for more than traffic filtering. Some of you may have heard of the eXpress Data Path (XDP) or the more recent eXpress Resubmission Path (XRP). Both of these are eBPF-powered shunts of kernel data paths that are used to optimize the system for very specific types of workloads. We'll return to these in a future lecture (and maybe a lab as well) since they can be considered advanced topics. For now, we'll focus on the third purpose eBPF can serve: execution tracing.

pwru is a tool created by Cilium to help trace network packets in the kernel's network stack and debug network connectivity issues. It does this by attaching simple eBPF programs to certain function entry points. These programs can report back to a userspace process different kinds of information, including the function that was reached, the arguments that were passed, and a CPU clock timestamp. The method used for instrumenting kernel code is based on kprobes. Ask your assistant for more information.

[10p] Task A - A packet's journey

Installation — build from source

Pre-built packages are no longer maintained for most distributions, so you'll build pwru from source. All you need is a Go compiler and make.

# Install Go if you don't have it
$ sudo apt install golang-go   # Ubuntu/Debian
# or follow https://go.dev/dl/ for the latest version
 
# Clone and build
$ git clone https://github.com/cilium/pwru.git
$ cd pwru
$ make
$ sudo mv pwru /usr/local/bin/
 
 
The build takes about a minute on first run (Go downloads dependencies). The result is a statically linked binary with no runtime dependencies.
 
**Minimum requirements** (check before running):
 
  * Linux kernel ≥ 5.5 (for BTF support): ''uname -r''
  * BTF enabled: ''ls /sys/kernel/btf/vmlinux''file must exist
  * ''bpf'' filesystem mounted: ''mount | grep bpf''
 
If BTF is missing, ''pwru'' will fail immediately with a clear error message.

Now, trace all outgoing DNS queries to the Google DNS (i.e.: 8.8.8.8) and perform one using dig. Add relative timestamps to the individual trace entries, to get an idea of the computational cost of each operation.

Finally, insert an iptables rule on the OUTPUT chain that drops DNS queries to 8.8.8.8 and redo the experiment. Check where the packet's path is cut short (the reason should be obvious :p).

Ubuntu users: local DNS caching via systemd-resolved may intercept your query before it reaches the network. If pwru shows nothing, try:

$ sudo systemd-resolve --flush-caches

or target 127.0.0.53 to confirm caching is the issue.

[20p] Task B - Interpreting the call path

Analyze the call path in the kernel network stack for the first scenario (when the packet actually made it out). Explain each step of the packet's journey.

Check out this map of the kernel subsystems, but note that the best source of information is always RTFS.

To structure your analysis, answer these questions in order:

  1. Where does the packet originate? Which function is the first to appear in the trace? What layer of the network stack does it correspond to?
  2. How does it reach the IP layer? Identify the transition from the socket/transport layer to the IP layer. Which function marks this boundary?
  3. What does Netfilter do here? Identify nf_hook_slow in the trace. Which Netfilter hook point does it correspond to (refer back to Figure 1 from Task 01)?
  4. How does it leave the machine? Identify the function responsible for handing the packet to the network device driver. What happens after this point?
  5. What changed with the DROP rule? Compare the two traces side by side. At which function does the path diverge?