Coder Legion - Recent posts tagged ebpf

Writing eBPF Probes in C for Advanced Kubernetes Observability

Tue, 13 Oct 2026 15:06:28 +0000

Traditional Kubernetes observability usually relies on sidecar proxies or application-level tracing, both of which introduce CPU and memory overhead. eBPF avoids this by running sandboxed C programs directly inside the Linux kernel, enabling zero-instrumentation tracing for syscalls, network events, and file platform operations.

Tracing TCP Connect Latency

Standard pod metrics usually miss what happens deep in the networking stack. By tracing tcp_v4_connect, we can measure exact TCP connection setup times across nodes without modifying application code or container images.

The eBPF C Program

The probe below is written in restricted C and compiled into eBPF bytecode using Clang/LLVM.

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

// Define a BPF map to store the start time of the connect call, keyed by PID
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, u64);
} start SEC(".maps");

// Kprobe triggered on entry to tcp_v4_connect
SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_enter, struct sock *sk)
{
    u64 ts = bpf_ktime_get_ns();
    u32 pid = bpf_get_current_pid_tgid() >> 32;

    bpf_map_update_elem(&start, &pid, &ts, BPF_ANY);
    return 0;
}

// Kretprobe triggered on exit of tcp_v4_connect
SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(tcp_v4_connect_exit, int ret)
{
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *tsp, delta_us;

    // Lookup the start time
    tsp = bpf_map_lookup_elem(&start, &pid);
    if (!tsp) {
        return 0; // Missed entry
    }

    // Calculate latency in microseconds
    delta_us = (bpf_ktime_get_ns() - *tsp) / 1000;
    
    // bpf_printk logs to /sys/kernel/debug/tracing/trace_pipe
    bpf_printk("PID %d TCP connect took %llu us, ret = %d\n", pid, delta_us, ret);

    bpf_map_delete_elem(&start, &pid);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Key Mechanics & Kernel Integration

#include <vmlinux.h>: Auto-generated from kernel BTF (BPF Type Format). It provides definitions for internal kernel structs without needing multiple kernel header files.
BPF_MAP_TYPE_HASH: eBPF programs cannot leverage global variables for dynamic state. Hash maps allow entry and exit probes to share timestamps across process IDs.
bpf_ktime_get_ns(): Fetches high-resolution monotonic timestamps from the kernel clock.
bpf_get_current_pid_tgid(): Returns the thread group ID in the upper 32 bits and thread ID in the lower 32 bits. Right-shifting by 32 extracts the user-space PID.
SEC("kprobe/...") / SEC("kretprobe/..."): Macros defining ELF section names. Loaders like libbpf read these to attach probes to kernel functions.

Deploying this probe via a Kubernetes DaemonSet alongside a user-space agent (for example, written in Go using cilium/ebpf) provides low-overhead network latency tracing across all pods on a node.

Originally published on anasrhimi.tech.

Building a Line-Rate Linux Kernel Firewall in Pure Rust with Aya and eBPF/XDP

Sat, 03 Oct 2026 13:29:33 +0000

Volumetric Distributed Denial of Service (DDoS) and transport-layer flooding attacks pose an existential challenge to modern cloud infrastructure, edge gateways, and container workloads.

Conventional Linux packet filtering utilities such as Netfilter (iptables and nftables) evaluate incoming frames only after the kernel has allocated a socket buffer (sk_buff) descriptor and scheduled softirq processing through the core TCP/IP stack. Under multi-gigabit line-rate flood conditions (millions of packets per second), the operating system spends the vast majority of its CPU cycles simply allocating and freeing sk_buff memory metadata, triggering kernel starvation and packet drop bottlenecks even before firewall rules are evaluated.

To address this fundamental OS-level bottleneck, I built xdp-guard: a production-grade, memory-safe in-kernel defense engine and per-IP rate limiter implemented strictly in Pure Rust using the Aya eBPF framework.

Why eBPF / XDP at the Driver Layer?

The eXpress Data Path (XDP) provides a programmable, bare-metal packet processing execution hook directly inside the Network Interface Card (NIC) driver subsystem before any memory descriptor allocation occurs.

Conventional Path:
[NIC RX] -> [IRQ / SoftIRQ] -> [Allocate sk_buff] -> [Netfilter/iptables] -> [Socket / App]
                                      ^
                           Severe CPU Allocation Overhead

xdp-guard Path:
[NIC RX] -> [eBPF / XDP Driver Hook] ---> (Match Blocklist) ---> [ XDP_DROP ] (Near-Zero CPU)
                             |
                      (Match Allowlist)
                             |
                             v
                   [ Pass to OS Stack ]

When hostile or unapproved packets are identified, returning XDP_DROP causes the network driver to immediately recycle the packet ring buffer without allocating a single byte of kernel socket memory.

Architectural Highlights

1. Zero C Dependencies (Pure Rust)

Traditional eBPF tools rely heavily on C toolchains (clang, llvm, libbpf). xdp-guard is built entirely in Rust from the user-space management CLI down to the no_std kernel eBPF bytecode using Aya.

2. Strict 64-Bit Memory Boundary Alignments

To ensure safe zero-copy communication across the kernel-userspace boundary, shared structures use explicit padding fields to guarantee 8-byte alignment:

#[repr(C)]
#[derive(Clone, Copy, Debug)]
pub struct RuleValue {
    pub action: u32,       // 0 = Pass, 1 = Drop
    pub _pad: u32,         // Explicit 4-byte padding for 8-byte boundary
    pub created_at: u64,   // Monotonic timestamp
    pub ttl_secs: u64,     // Time-to-live expiration
}

The presence of _pad: u32 is critical: it prevents architecture-specific compiler padding discrepancies and satisfies the Linux Kernel BPF Verifier.

3. Dynamic BPF Virtual Filesystem (bpffs) Synchronization

Firewall rules (blocklist and allowlist) are dynamically synchronized using pinned BPF hash maps located in /sys/fs/bpf/xdp_guard. Rules can be inserted or deleted instantaneously from the CLI without tearing down network interfaces or interrupting traffic flows.

4. In-Kernel Token-Bucket Rate Limiter

In addition to static blocklists, xdp-guard embeds a stateful token-bucket rate limiter directly in kernel space. Monotonic arrival intervals ($\Delta t = t{now} - t{last}$) are computed using nanosecond bpf_ktime_get_ns() clocks, discarding over-limit burst floods at line-rate.

Empirical Verification: Real Linux Kernel Telemetry

To validate xdp-guard under realistic operating conditions, we set up an automated Linux network namespace testbed over virtual ethernet (veth) pairs (xdp_client at 10.10.0.2 and xdp_server at 10.10.0.1).

Telemetry streamed directly from the kernel STATS_MAP array demonstrated instantaneous 100.00% packet mitigation upon inserting an attacker IP into the kernel blocklist, followed by zero-loss recovery:

TIMESTAMP            PASSED (PKTS)   DROPPED (PKTS)  DROP RATIO     
-----------------------------------------------------------------
09:56:56 | Pass: 1        (30.2 KB) | Drop: 0        (0 B)   | Ratio: 0.00%
[!] Inserting IP 10.10.0.2 into BLOCKLIST_MAP... [SUCCESS]
09:56:57 | Pass: 0        (30.2 KB) | Drop: 1        (98 B)  | Ratio: 100.00%
09:56:58 | Pass: 0        (30.2 KB) | Drop: 1        (196 B) | Ratio: 100.00%
09:56:59 | Pass: 0        (30.2 KB) | Drop: 1        (294 B) | Ratio: 100.00%
...
09:57:55 | Pass: 0        (30.2 KB) | Drop: 1        (5.7 KB)| Ratio: 100.00%
[!] Removing IP 10.10.0.2 from BLOCKLIST_MAP... [SUCCESS]
09:57:56 | Pass: 1        (30.3 KB) | Drop: 0        (5.7 KB)| Ratio: 0.00%
09:57:57 | Pass: 1        (30.4 KB) | Drop: 0        (5.7 KB)| Ratio: 0.00%

Every single drop occurred directly within the NIC driver hook, requiring zero kernel TCP/IP stack overhead.

Source Code & Research Paper

The complete project, documentation, and formal research paper are fully open-source:

GitHub Repository: https://github.com/rahadbhuiya/xdp-guard
Citable Zenodo DOI: 10.5281/zenodo.23120995
Automated CI: Continuous testing across Rust Stable and Nightly LLVM targets via GitHub Actions.

If you are working on Linux kernel systems, eBPF, or Rust networking, I would love to hear your thoughts, feedback, or suggestions!

Booting ARM64 Linux in my Rust KVM Hypervisor: Getting to the console with help of eBPF

Fri, 25 Sep 2026 14:40:06 +0000

Recap

In the previous article I got ARM64 Linux as far
as executing inside my Rust KVM hypervisor. I still couldn't actually interact
with the guest, though: I had no working console.

This article is about getting from "the PC is moving" to typing "hello" into
Linux and getting "HELLO" back.

UART

Last time we learned how to obtain the device tree blob from QEMU's virt
machine. Now we want to take a look at what's inside. We can decompile it:

dtc -I dtb -O dts -o virt.dts virt.dtb

Then, a simple search through the file already gives us the clue we need:

pi@raspberrypi:~ $ grep -i uart virt.dts -A5 -B1
        pl011@9000000 {
                clock-names = "uartclk\0apb_pclk";
                clocks = <0x8000 0x8000>;
                interrupts = <0x00 0x01 0x04>;
                reg = <0x00 0x9000000 0x00 0x1000>;
                compatible = "arm,pl011\0arm,primecell";
};

So, now I had a lead: I needed to emulate the pl011 UART at 0x09000000 so
that the existing Linux pl011 driver could communicate with it. This was not a
really interesting part, as most of it was simply reading Linux kernel source
code along with pl011 documentation from ARM. Soon I was able to get output
from Linux and it felt really great:

[    0.000000] Linux version 7.1.3 (docker@stjepan-Aspire-F5-573G) (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT Thu Jul 16 08:52:24 UTC 2026
[    0.000000] KASLR disabled on command line
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: UEFI not found.
[    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x0000000047ffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x47fc10c0-0x47fc37bf]
                  ### omitted further kernel log ###

Creating custom initramfs

Now I needed bidirectional UART support so that I could send input to the guest
as well. To make further investigation easier, I created a very simple binary
written in C to be run as the Linux init process: it just prints out current
interrupt statistics and asks user for input, echoing it back in all caps. All
we then need is initramfs.txt file with some simple folders and devices:

dir / 0755 0 0
dir /dev 0755 0 0
dir /proc 0555 0 0
dir /sys 0555 0 0
nod /dev/kmsg 0600 0 0 c 1 11
nod /dev/console 0600 0 0 c 5 1
file /init /home/stjepanp/Develop/initramfs/init 0755 0 0

I only needed enough of a filesystem to provide /init, /dev/console, and
access to /proc for interrupt statistics. Then, in the Linux source code
folder, we generate initramfs with:

./usr/gen_initramfs.sh ../initramfs/initramfs.txt -o initramfs

To see at which memory address to add initramfs, we can run ordinary QEMU with
our initramfs and see what happens:

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt,dumpdtb=virt.dtb                                \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0 nokaslr rdinit=/init"          \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults                                             \
        -initrd "./initramfs"

After decompiling virt.dtb (explained in previous article) we can see the location:

pi@raspberrypi:~/dtb $ grep initrd virt-initramfs.dts -B2 -A3
        chosen {
                bootargs = "console=ttyAMA0 initramfs_async=0 rdinit=/init nokaslr";
                linux,initrd-start = <0x0 0x40200000>;
                linux,initrd-end = <0x0 0x40294200>;
                stdout-path = "/pl011@9000000";
                kaslr-seed = <0x8114b902 0xe815aa93>;
        };

We do have to recalculate initrd-end if size of our initramfs file changes.
If that happens, we can easily recompile the device tree blob after changing
the value. Also, let's not forget to load initramfs in Rust code:

const LOAD_ADDR: u64 = 0x40_000_000;
const INITRAMFS_OFFS: u64 = 0x200_000;
const KERN_OFFS: u64 = 0x200_000 + INITRAMFS_OFFS;

impl VM {
    pub fn arch_load_linux(&mut self, vcpu: &mut VCPU, args: &Args) -> io::Result<()> {
        let linux_mem_idx = self.add_mem_region(1024 * 1024 * 1024, LOAD_ADDR)?;
        self.load_file_to_memory(linux_mem_idx, &dtb_path, 0x0)?;
        self.load_file_to_memory(linux_mem_idx, &initramfs_path, INITRAMFS_OFFS)?;
// ...
    }

Setting up vGIC

Even without having the ability to send keystrokes there was one big problem
with the Linux console stdout: The kernel messages were reaching the console,
but once execution reached /init, output stopped behaving as expected. It
took me some time to realize that the ARM architectural timer was also involved.
Its interrupts need to be delivered through the virtual GIC, so I first needed
to create a vGIC. It turned out creating the vGIC was easy. I just followed the
logic from QEMU source code:

https://elixir.bootlin.com/qemu/v11.0.1/source/accel/kvm/kvm-all.c#L4035

In Rust this translates to:

let mut create_device: kvm_create_device = unsafe { std::mem::zeroed() };
create_device.type_ = KVM_ARM_DEVICE_VGIC_V2;
create_device.fd = 0xffffffff;
create_device.flags = 0;

unsafe { libc::ioctl(self.fd, KVM_CREATE_DEVICE, &mut create_device); }

This worked. Also note that I have omitted proper error handling to keep the
code short and concise.

Tracing IRQ injection

Now if we run our initramfs with strace as before, and we try to type in
some keys, we will see that in our log we have:

pi@raspberrypi:~ $ grep -i irq kvm.log | tail -n5
458398 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7fd932dbd0) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db30) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db20) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0

So our ioctl for IRQ injection actually takes an address as argument. If we
investigate this, we can find out how KVM_IRQ_LINE is handled in the Linux
kernel:

https://elixir.bootlin.com/linux/v6.1.21/source/virt/kvm/kvm_main.c#L4776

Reading this code we see that the relevant function is kvm_vm_ioctl_irq_line
that takes struct kvm_irq_level as second argument and this is what we need
to interpret:

/* for KVM_IRQ_LINE */
struct kvm_irq_level {
/*
 * ACPI gsi notion of irq.
 * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47..
 * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23..
 * For ARM: See Documentation/virt/kvm/api.rst
 */
union {
__u32 irq;
__s32 status;
};
__u32 level;
};

For this purpose, we can treat it as an 8-byte structure containing the IRQ number and level. Therefore, we can easily inspect it again via eBPF when we
run QEMU with our initramfs:

pi@raspberrypi:~ $ sudo bpftrace -e 'kprobe:kvm_vm_ioctl_irq_line { printf("irq_level=%x\n", *((uint64 *)arg1)); }'
Attaching 1 probe...
irq_level=1000021
irq_level=1000021
irq_level=1000021
irq_level=1000021

Without wasting much time we can easily reconstruct this behavior in Rust:

pub fn arch_update_irq(level: u32, vm_fd: libc::c_int) -> io::Result<()> {
    let mut irq: kvm_irq_level = unsafe { std::mem::zeroed() };
    irq.level = level;
    irq.__bindgen_anon_1.irq = 0x1000021;

    let ret = unsafe {
        libc::ioctl(vm_fd, KVM_IRQ_LINE, &mut irq)
    };

    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    Ok(())
}

Note that, in essence, 0x1000021 is not an arbitrary magic number. On ARM64,
KVM_IRQ_LINE encodes the interrupt type and interrupt ID in the irq field. Here 0x01 selects an SPI and 0x21 is interrupt 33 the UART interrupt from
the device tree.

No such device or address

However, when we run our program, it fails with "No such device or address". To
get some clue on what is going on, we can try strace and get:

ioctl(5, KVM_RUN, 0)                    = -1 ENXIO (No such device or address)
Error: Os { code: 6, kind: Uncategorized, message: "No such device or address" }

My first impulse was to search for ENXIO in Elixir bootlin, but,
unfortunately, there were simply too many functions and branches that could
return ENXIO. Rather than trying to follow every possible ENXIO path
manually, I decided to use a more brute-force approach. I attached kretprobes
to the KVM functions and looked for one returning -ENXIO:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:kvm_* /retval == (uint32)(-6)/ { printf("%s\n", probe); }'
Attaching 487 probes...
kretprobe:kvm_vgic_map_resources
kretprobe:kvm_arch_vcpu_run_pid_change

The second function here does not seem really relevant, so I tried looking for
kvm_vgic_map_resources on Elixir:

https://elixir.bootlin.com/linux/v6.1.21/source/arch/arm64/kvm/vgic/vgic-v2.c#L289

Now taking a look at the function we can clearly see where ENXIO is returned:

int vgic_v2_map_resources(struct kvm *kvm)
{
    struct vgic_dist *dist = &kvm->arch.vgic;
    int ret = 0;

    if (IS_VGIC_ADDR_UNDEF(dist->vgic_dist_base) ||
        IS_VGIC_ADDR_UNDEF(dist->vgic_cpu_base)) {
        kvm_debug("Need to set vgic cpu and dist addresses first\n");
        return -ENXIO;
    }
    // ...

Now we have a hint. We need to set vGIC CPU and distributor addresses first. To
confirm our hypothesis we can try tracing this function specifically:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:vgic_v2_map_resources { printf("ret=%u\n", retval); }'
Attaching 1 probe...
ret=4294967290

Finding out distributor address

My first attempt to set up distributor address didn't go well. I wrote the
following code (I extracted distributor address from the device tree and
tried to reconstruct the logic from QEMU source code):

let dist_addr: u64 = 0x08_000_000;
let mut dev_attr: kvm_device_attr = unsafe { std::mem::zeroed() };
dev_attr.group = KVM_DEV_ARM_VGIC_GRP_ADDR;
dev_attr.attr = KVM_VGIC_V2_ADDR_TYPE_DIST;
dev_attr.flags = 0;
dev_attr.addr = dist_addr;

unsafe { libc::ioctl(create_device.fd as i32, KVM_SET_DEVICE_ATTR, &mut dev_attr) };

However, when I ran the program, I got:

ioctl(6, KVM_SET_DEVICE_ATTR, 0x7ffe2696d8) = -1 EFAULT (Bad address)

To debug this, I needed to inspect the values defined in
struct kvm_device_attr, but it was a lot more complicated: I had to check if
the struct I was tracing was for KVM_DEV_ARM_VGIC_GRP_ADDR group and for
KVM_VGIC_V2_ADDR_TYPE_DIST. So, I decided to write my eBPF program in C with
and called it probe-kvm.bpf.c:

#include <linux/bpf.h>
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <linux/kvm.h>
#include <stdbool.h>
#include <stdint.h>

char LICENSE[] SEC("license") = "GPL";

SEC("kprobe/vgic_v2_set_attr")
int BPF_KPROBE(kprobe_vgic_v2_set_attr, void *dev, struct kvm_device_attr *dev_attr)
{
        struct kvm_device_attr attr;
        __u64 value;

        bpf_probe_read_kernel(&attr, sizeof(attr), dev_attr);
        if (attr.group != KVM_DEV_ARM_VGIC_GRP_ADDR) {
                return 0;
        }

        if ((attr.attr != KVM_VGIC_V2_ADDR_TYPE_DIST) && (attr.attr != KVM_VGIC_V2_ADDR_TYPE_CPU)) {
                return 0;
        }

        bpf_printk("\nset_attr: attr=%s addr=0x%llx", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr);

        return 0;
}

To compile and run it I wrote the following script:

#!/bin/sh

clang -O2 -g -mcpu=v1 -target bpf -I /usr/include/aarch64-linux-gnu -c probe-kvm.bpf.c -D__TARGET_ARCH_arm64 -o probe-kvm.bpf.o
sudo rm -rf /sys/fs/bpf/vgic
sudo bpftool prog loadall probe-kvm.bpf.o /sys/fs/bpf/vgic autoattach
sudo cat /sys/kernel/debug/tracing/trace_pipe

So, I started ordinary QEMU along my eBPF trace and got this output:

$ ./start-probe.sh
           <...>-146994  [002] d..31 953951.139387: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fd4d3c450

           <...>-146994  [002] d..31 953951.139418: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fd4d3c450

This really looked like a userspace address while I was supplying a raw address.
To find out what was at that address I modified the last line of the BPF program
above:

        if (bpf_probe_read_user(&value, sizeof(value), (void *)attr.addr) != 0)
                return -1;

        bpf_printk("\nset_attr: attr=%s addr=0x%llx, *addr=0x%llx\n", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr, value);

With this I got what I didn't really expect:

           <...>-147143  [000] d..31 954397.183145: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fdc806ca0, *addr=0x8010000

           <...>-147143  [000] d..31 954397.183177: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fdc806ca0, *addr=0x8000000

So the addr field is actually a pointer to the address, and not the address
itself. Note that 0x8010000 and 0x8000000 are the guest physical
addresses of the GIC CPU interface and distributor taken from the virt machine's
device tree:

pi@raspberrypi:~/dtb $ grep -i intc virt.dts -A3
        intc@8000000 {
                phandle = <0x8001>;
                reg = <0x00 0x8000000 0x00 0x10000 0x00 0x8010000 0x00 0x10000>;
                compatible = "arm,cortex-a15-gic";

This also explained the EFAULT: KVM was trying to dereference
attr.addr as a userspace pointer. With this small fix, I got a fully working
Linux console:

dev_attr.addr = &dist_addr as *const u64 as u64;

The following snippet shows that my init process (as part of
initramfs) was run and it echoed back my input in capital letters:

[    0.832623] clk: Disabling unused clocks
[    0.834046] PM: genpd: Disabling unused power domains
[    0.845565] Freeing unused kernel memory: 12672K
[    0.847475] Run /init as init process
           CPU0       
 11:        199 GIC-0  27 Level     arch_timer
 13:          0 GIC-0  33 Level     uart-pl011
IPI0:         0       Rescheduling interrupts
IPI1:         0       Function call interrupts
IPI2:         0       CPU stop interrupts
IPI3:         0       CPU stop NMIs
IPI4:         0       Timer broadcast interrupts
IPI5:       168       IRQ work interrupts
IPI6:         0       CPU backtrace interrupts
IPI7:         0       KGDB roundup interrupts
Err:          0

type a string: hello
allcaps: HELLO

[    2.224776] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    2.227382] CPU: 0 UID: 0 PID: 1 Comm: init Not tainted 7.1.3 #1 PREEMPT 
[    2.229749] Hardware name: linux,dummy-virt (DT)

The interrupt statistics are also printed from my init. Also, when init
exits, we get the (correct) Kernel panic that init process was terminated.

Final thoughts

At this point I finally had what I was looking for: a real Linux userspace
running inside my own KVM-based hypervisor, with a working UART, interrupt
delivery and a minimal initramfs.

More importantly, I now had a way to interact with the guest and inspect what
was happening inside KVM. That gives me a much better foundation for the next
step: implementing more of the virtual hardware needed to turn this into a
useful ARM64 virtual machine.

You can also check out the repository on my GitHub page and, if you have a
Raspberry Pi 4B, try the VMM yourself:

https://github.com/StjepanPoljak/kvm-rust

The Invisible Journey: Tracing Packets Through the Linux Kernel with eBPF and `pwru`

Sun, 26 Jul 2026 04:00:42 +0000

The modern developer's landscape is a complex tapestry of microservices, containers, and cloud infrastructure. When an application slows down, or a connection drops, the immediate culprits are often assumed to be the application code itself or the external network. But what about the invisible journey a packet takes inside the operating system? That often-opaque realm, the Linux kernel, holds the true story of network performance, packet drops, and elusive latency spikes.

For many developers, the kernel remains a mysterious black box. We interact with it daily through syscalls and network interfaces, but rarely peer into its inner workings. Traditional tools like tcpdump and netstat offer glimpses – what leaves our machine, what arrives, and what ports are open – but they often fall short when we need to understand why a packet was dropped, where it spent its time, or how it was processed before it ever reached the application. This is the gap that kernel-level observability tools, powered by eBPF, are rapidly filling.

Inspired by the journey of a fellow developer learning to navigate the kernel's network stack, this article aims to demystify the process of tracking a packet. We'll explore why deep kernel insight is crucial for modern debugging, introduce the revolutionary technology of eBPF, and then dive hands-on into pwru – an eBPF-based tool that makes the invisible journey of a packet through the Linux kernel visible and understandable. You don't need to be a kernel developer to unlock these insights; you just need the right tools and a guide to interpret their output.

The Invisible Journey: Why Deep Packet Tracing Matters

Imagine a web request taking too long. Your application logs show the request was received, but the response was delayed. Is it the database? The application logic? Or something happening deep within the network stack of your server, or even the client? Without kernel-level visibility, you're often left guessing.

Beyond the Wire: The Limitations of Userspace Tools

Tools like tcpdump are invaluable. They capture packets as they hit or leave a network interface, showing you what's on the wire. netstat or ss show you active connections, listening ports, and basic statistics. These are the frontline tools for network debugging, and they serve a critical purpose.

However, they have inherent limitations when you need to understand the internal processing of a packet:

Black Box Kernel: tcpdump sees packets before the kernel processes them on ingress and after the kernel has finished processing them on egress. What happens in between – routing decisions, firewall rules, socket buffer management, TCP stack processing – remains a mystery.
Packet Drops: If tcpdump doesn't show a packet leaving your machine, you know it never made it to the wire. But why? Was it dropped by a firewall? A full transmit queue? A routing error? tcpdump can't tell you the precise point of failure within the kernel.
Latency Attribution: A high round-trip time could be due to network latency, but it could also be due to significant processing delays within the kernel's network stack or the application's inability to read from the socket quickly enough. Isolating these factors is hard with userspace tools alone.
Context: Userspace tools often lack the rich context available inside the kernel, such as the specific kernel function being executed, the CPU core involved, or the network namespace.

Common Scenarios Demanding Kernel-Level Insight

Developers frequently encounter scenarios where kernel-level packet tracing becomes indispensable:

Mysterious Packet Drops: "My service isn't receiving all the packets it expects, but tcpdump on the interface shows them arriving." This suggests an internal kernel drop.
Unexplained Latency: "My application is slow, but CPU and memory usage are fine, and network latency to other services seems normal." Could it be contention within the kernel's network buffers or inefficient processing?
Firewall Troubleshooting: "I've configured iptables, but traffic still isn't flowing as expected, or is being unexpectedly blocked." Pinpointing the exact netfilter hook where a packet is dropped or modified is crucial.
Routing and Forwarding Issues: "Why isn't this packet being forwarded to the correct destination, even though my routing table looks correct?" Tracing the packet's path through the routing decision points can reveal the answer.
Complex Network Stacks: In environments using technologies like IPVS, BPF-based load balancers, or advanced container networking (e.g., Cilium, Calico), understanding the kernel's internal packet flow becomes paramount.

These challenges highlight the need for a deeper level of observability – one that allows us to peek inside the kernel and follow a packet's every step.

Enter eBPF: Programmable Kernel Observability

For decades, getting insight into the Linux kernel involved recompiling it, using ftrace, or relying on limited procfs entries. These methods were often cumbersome, risky, or lacked the necessary detail. The advent of eBPF (extended Berkeley Packet Filter) changed everything.

What is eBPF? (A High-Level Overview)

At its core, eBPF is a revolutionary technology that allows programs to run safely within the Linux kernel. Think of it as a highly efficient, event-driven virtual machine embedded directly in the kernel. Unlike traditional kernel modules, eBPF programs are:

Safe: All eBPF programs must pass a rigorous kernel verifier before execution, ensuring they don't crash the kernel, loop indefinitely, or access unauthorized memory.
Event-Driven: eBPF programs are attached to various "hook points" within the kernel. These can be:
- Kprobes/Kretprobes: Attach to the entry or exit of almost any kernel function.
- Uprobes/Uretprobes: Attach to the entry or exit of userspace functions.
- Tracepoints: Stable, well-defined points intentionally added by kernel developers for tracing.
- Network Hooks: For filtering and processing network packets.
- Syscall Hooks: Intercepting system calls.
Efficient: They execute in-kernel, avoiding costly context switches between userspace and kernel space. They are typically JIT-compiled to native machine code for maximum performance.
Programmable: Developers write eBPF programs (often in C, then compiled to eBPF bytecode using clang and llvm) to perform specific tasks: filter events, collect statistics, modify data, or redirect packets. These programs can then communicate results back to userspace via shared maps or ring buffers.

In essence, eBPF provides a powerful, flexible, and safe way to extend kernel functionality and gain unprecedented visibility into its operations without modifying kernel source code or rebooting.

The Power of eBPF for Networking

eBPF's origins are deeply rooted in networking (the original BPF was for packet filtering). Today, its capabilities for network observability and control are immense:

Advanced Packet Filtering: Beyond what iptables can do, eBPF programs can filter packets based on arbitrary criteria, even deep into application-layer protocols.
Traffic Control: Shaping, policing, and load balancing traffic directly in the kernel.
Security: Implementing fine-grained network policies and intrusion detection.
Observability: This is where pwru shines. By attaching eBPF programs to key network functions and tracepoints, we can collect detailed information about every packet as it traverses the kernel's network stack. This allows us to see exactly what the kernel is doing with each packet, providing unparalleled debugging capabilities.

`pwru`: Your Guide Through the Kernel Network Stack

While you could write your own eBPF programs to trace packets, tools like pwru (Packet Walk Ruler) abstract away the complexity, providing a ready-to-use solution for deep network stack inspection. pwru leverages eBPF to dynamically attach to numerous kernel functions and tracepoints, giving you a real-time, step-by-step view of a packet's journey.

Introducing `pwru`

pwru isn't just another tcpdump. It's a specialized tool designed to show you:

Hook Point: The specific kernel function or tracepoint the packet is currently passing through.
Action: What the kernel did with the packet at that point (e.g., PASS, CONSUMED, DROP, QUEUE).
Packet Metadata: Crucial details like source/destination IP/port, protocol, packet length (skb_len), network namespace, CPU, and more.
Context: Which netfilter hook, if any, was involved, and the return code.

This level of detail allows you to precisely identify where packets are being handled, modified, or dropped within the kernel, making it an indispensable tool for debugging complex network issues.

Setting Up `pwru` (Practical Steps)

pwru requires a relatively recent Linux kernel (4.9+ for basic eBPF, 5.x+ for full features and stability) and a few development tools to compile its eBPF programs.

Prerequisites:
- Linux Kernel: Version 4.9+ (5.x or newer recommended).
- Build Tools: clang, llvm, make, git.
- Kernel Headers: Ensure your kernel headers match your running kernel version. On Debian/Ubuntu, sudo apt install linux-headers-$(uname -r). On Fedora/CentOS/RHEL, sudo yum install kernel-devel.

Installation:
pwru is typically installed by cloning its GitHub repository and building it.

# Ensure you have git, clang, llvm, and make installed
sudo apt update && sudo apt install git clang llvm make   # For Debian/Ubuntu
# sudo yum install git clang llvm make                   # For Fedora/CentOS/RHEL

git clone https://github.com/cilium/pwru.git
cd pwru
make
sudo make install

If make install fails or you prefer not to install globally, you can run pwru directly from the pwru/bin directory.

Your First Packet Trace with `pwru`

Let's start with a simple example: tracing an HTTP request to a local web server (or any external website).

Open two terminal windows. In the first, start pwru:

# Terminal 1: Start pwru
# Trace TCP packets on port 80 (HTTP) or 443 (HTTPS)
# We'll customize output fields for clarity
sudo pwru --output-fields=src_ip,dst_ip,proto,port,func,action,skb_len \
          --filter "proto == IPPROTO_TCP && (dst_port == 80 || dst_port == 443)"

In the second terminal, make an HTTP request:

# Terminal 2: Make an HTTP request
curl -v http://example.com
# Or, if you have a local web server:
# curl -v http://localhost

Back in Terminal 1, you'll see a flood of output. Let's break down a typical line:

src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=ip_rcv action=PASS skb_len=60
src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=ip_local_deliver_finish action=PASS skb_len=60
src_ip=192.168.1.100 dst_ip=93.184.216.34 proto=TCP port=51234->80 func=tcp_v4_rcv action=PASS skb_len=60
... (many more lines) ...

What are we seeing? Each line represents a "hook point" – a specific kernel function or tracepoint – that the packet traversed.

src_ip, dst_ip, proto, port: Basic network tuple identifying the packet.
func: The kernel function name (ip_rcv, tcp_v4_rcv, etc.). This is the heart of pwru's insight.
action: What happened at that function. PASS means it continued its journey. DROP would indicate it was discarded. CONSUMED often means it was processed and removed from the general network path (e.g., by the TCP stack).
skb_len: The size of the sk_buff (socket buffer) structure carrying the packet data.

This output shows the packet moving through different layers and processing stages within the kernel. For an outgoing packet, you'd see functions like ip_output, dev_queue_xmit, etc. For an incoming packet destined for a local socket, you'd see ip_rcv, ip_local_deliver, tcp_v4_rcv, and eventually sock_queue_rcv_skb.

Deeper Dives: Understanding the Kernel Network Path

Interpreting pwru's output requires a basic understanding of how the Linux kernel processes network packets. While the full network stack is incredibly complex, we can identify key functions and their roles.

Key Kernel Hook Points and Their Significance

The kernel network stack can be broadly divided into ingress (receiving) and egress (transmitting) paths.

Ingress Path (Receiving a Packet):

__netif_receive_skb_core: The initial entry point for a packet received from the network driver.
ip_rcv: The entry point for IPv4 packets after __netif_receive_skb_core. It performs basic validation and determines if the packet is for the local host or needs forwarding.
ip_local_deliver_finish: If the packet is for the local host, this function is called. It passes the packet to the appropriate transport layer (TCP, UDP, ICMP).
tcp_v4_rcv / udp_rcv: Entry points for TCP and UDP packets, respectively, after ip_local_deliver_finish. These functions handle protocol-specific processing (e.g., TCP sequence number checks, ACK generation).
sock_queue_rcv_skb: The packet is finally queued to the receiving socket's buffer, making it available for a userspace application to read via recvmsg, read, etc.

Egress Path (Transmitting a Packet):

tcp_sendmsg / udp_sendmsg: Initiated by a userspace application's sendmsg or write call. These prepare the packet for transmission at the transport layer.
ip_output: Handles the IPv4 header creation and prepares the packet for sending.
ip_finish_output: Finalizes IP output, including routing decisions.
dev_queue_xmit: Queues the packet to the appropriate network device's transmit queue.
__dev_xmit_skb: The packet is handed off to the network driver for physical transmission.

pwru shows you the packet traversing many of these functions (and many more in between), along with netfilter hooks (NF_HOOK points) and other processing stages.

Common Scenarios & `pwru` Insights

Let's look at how pwru can help diagnose specific problems.

1. Pinpointing Packet Drops

One of the most powerful uses of pwru is identifying where and why a packet is dropped. If you suspect packet loss:

# Terminal 1: Watch for dropped packets
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len --filter "action == DROP"

Now, try to reproduce the packet loss (e.g., attempt a connection to a non-existent port on your machine, or trigger a firewall rule). You might see output like:

func=ip_rcv_finish action=DROP src_ip=192.168.1.1 dst_ip=192.168.1.100 proto=ICMP port=0->0 skb_len=84
func=nf_hook_slow action=DROP src_ip=192.168.1.1 dst_ip=192.168.1.100 proto=TCP port=12345->80 skb_len=60

ip_rcv_finish with DROP might indicate issues like a full receive queue or an invalid packet.
nf_hook_slow with DROP is a strong indicator that a netfilter (iptables) rule is dropping the packet. This immediately tells you to inspect your firewall configuration.

2. Analyzing Latency Attribution

If curl to localhost is slow, where is the time spent? pwru can show the timestamps for each hook point, though interpreting the exact delta requires careful analysis and often correlation with other tools. For a quick check, you can add ktime (kernel time in nanoseconds) to your output:

# Terminal 1: Trace a local request with timestamps
sudo pwru --output-fields=ktime,func,action,src_ip,dst_ip,proto,port,skb_len \
          --filter "proto == IPPROTO_TCP && (src_ip == 127.0.0.1 || dst_ip == 127.0.0.1) && (dst_port == 80 || src_port == 80)"

Then, in Terminal 2: curl http://localhost.

By examining the ktime values, you can see if there are unusually large gaps between consecutive func calls for the same packet. A long gap between ip_rcv and ip_local_deliver could suggest CPU contention or heavy processing before the packet reaches the transport layer. A delay between sock_queue_rcv_skb and the application's actual read might point to application-level slowness.

3. Understanding Firewall/Netfilter Interaction

pwru can explicitly show netfilter processing. netfilter hooks are points where iptables rules operate.

# Terminal 1: Trace packets, specifically looking for netfilter hooks
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len,hook_name,nf_verdict \
          --filter "proto == IPPROTO_TCP && (dst_port == 22 || src_port == 22)" # e.g., SSH traffic

You'll see lines like:

func=nf_hook_slow action=PASS hook_name=NF_INET_PRE_ROUTING nf_verdict=NF_ACCEPT src_ip=...
func=nf_hook_slow action=PASS hook_name=NF_INET_LOCAL_IN nf_verdict=NF_ACCEPT src_ip=...

hook_name: Specifies the netfilter hook (e.g., NF_INET_PRE_ROUTING, NF_INET_LOCAL_IN, NF_INET_FORWARD).
nf_verdict: The verdict of the netfilter chain (NF_ACCEPT, NF_DROP, NF_STOLEN, etc.).

If a packet is dropped by netfilter, you'll see an action=DROP along with nf_verdict=NF_DROP at a specific hook_name, telling you exactly which iptables chain is responsible.

4. Tracing Specific Connections

To focus on a single problematic connection, use detailed filters:

# Terminal 1: Trace a specific source IP and destination port
sudo pwru --output-fields=func,action,src_ip,dst_ip,proto,port,skb_len \
          --filter "src_ip == 10.0.0.5 && dst_port == 8080"

This ensures you only see the relevant packets, reducing noise and making analysis much easier. You can combine filters with AND, OR, NOT for very precise targeting.

Advanced `pwru` Techniques and Trade-offs

pwru is a powerful tool, and mastering its capabilities involves more than just basic tracing.

Filtering and Output Customization

Complex Filters: pwru supports complex logical expressions in its --filter argument.

# Trace HTTP and HTTPS traffic, but only from a specific source IP, and not if it's an ICMP packet
sudo pwru --filter "proto == IPPROTO_TCP && (dst_port == 80 || dst_port == 443) && src_ip == 192.168.1.10 AND NOT proto == IPPROTO_ICMP"

Custom Output Fields: The --output-fields argument is crucial. You can add fields like:
- cpu: Which CPU core processed the packet. Useful for identifying CPU affinity issues.
- ktime: Kernel timestamp (nanoseconds).
- netns: Network namespace ID. Essential in containerized environments.
- ifindex: Network interface index.
- mark: skb->mark value, often used for policy routing or firewall marking.
- tstamp: Hardware timestamp.
- ingress_ifindex, egress_ifindex: The interface the packet entered/exited.
- reason: Provides more context for drops.
Consult pwru --help for a full list of available fields.
Saving Output: Redirect pwru's output to a file for later analysis, especially for long-running traces.
```
sudo pwru --output-fields=... --filter "..." > pwru_trace.log
```

Performance Considerations

eBPF is designed for low overhead, but pwru attaches to many kernel hook points. On a very busy system with high packet rates, tracing every packet can still generate a substantial amount of data and consume CPU cycles.

Precision is Key: Always use the most precise filters possible. Instead of --filter "proto == IPPROTO_TCP", use proto == IPPROTO_TCP && (src_port == 12345 || dst_port == 8080) to focus on relevant traffic.
Output Fields: Only request the output fields you truly need. Each field adds a small amount of overhead.
Short Bursts: For initial diagnosis, run pwru for short bursts (e.g., 5-10 seconds) to capture a representative sample of traffic during a problem occurrence, rather than letting it run indefinitely.
Production Caution: While generally safe, running pwru on highly loaded production systems without careful filtering and monitoring should be approached with caution. Test your filters in a staging environment first.

Limitations and Alternatives

While pwru is incredibly powerful, it's not a silver bullet:

Kernel Knowledge Still Required: Interpreting the output, especially the func names, still requires some understanding of the kernel's network stack. You won't become a kernel expert overnight, but you'll learn key functions relevant to your debugging tasks.
Root Privileges: pwru requires root privileges to attach eBPF programs to the kernel.
Kernel Version Dependency: While pwru tries to be compatible, specific hook points or data structures can vary slightly between kernel versions, potentially affecting output or functionality.
Not a Packet Capture Tool: pwru shows kernel events, not the full packet payload. For payload inspection, tcpdump or Wireshark are still necessary.

Alternatives and Complementary Tools:

BCC Tools: The BPF Compiler Collection (BCC) provides a rich set of eBPF-based tools for various aspects of kernel observability, including networking (e.g., tcplife, dropwatch, sockstat). They are often more specialized than pwru.
bpftrace: If pwru's predefined tracing isn't enough, bpftrace allows you to write custom eBPF one-liners or scripts with a simpler syntax than raw C. This is for when you need to combine specific kernel events or perform custom aggregations.
tracepoint-analyzer: Another tool specifically designed to help understand kernel tracepoints.

pwru serves as an excellent starting point and often provides enough detail for most network debugging scenarios. When you hit its limits, bpftrace or specialized BCC tools are the next logical step.

Conclusion

The journey of a packet through the Linux kernel is often an invisible one, a black box that can hide the root causes of frustrating network performance issues, mysterious packet drops, and elusive latency. For too long, developers have been forced to debug these problems with limited visibility, relying on guesswork and external network tools that only show part of the picture.

eBPF has fundamentally changed this landscape, empowering us to safely and efficiently peer into the kernel's inner workings. Tools like pwru democratize this power, translating complex kernel events into actionable insights that even developers "who barely know the kernel" can understand and leverage.

By embracing pwru and its eBPF underpinnings, you gain a superpower: the ability to trace a packet's every step, understand its fate, and pinpoint precisely where things go wrong within the operating system. This deeper understanding