Coder Legion - Recent posts tagged kvm

Notes on x86 PVH boot

Fri, 25 Sep 2026 14:47:35 +0000

Motivation

My Rust KVM project now being able to boot Linux on ARM64, I decided to try and
boot Linux on x86. I have been trying in the past to set up the boot procedure
via boot_params struct. At the time it was a lot more complex and now I wanted
to have some result as soon as possible. So I turned to the PVH
(Para-Virtualized Hardware) Linux boot which simplifies this process a lot. As
a Xen Project protocol, PVH provides a relatively simple interface between a
VMM and the Linux kernel, without requiring the VMM to reproduce the
traditional firmware/bootloader path.

Entering protected mode via BIOS

The traditional way of booting an operating system on an x86 machine is to start
in real mode and then proceed to protected mode. From there, if CPU is 64-bit,
we can go to long mode. In the PVH Linux entry path, Linux performs the
transition from the required 32-bit protected-mode entry state into 64-bit mode
itself.

Setting up real mode

x86 starts at reset vector (physical address 0xFFFFFFF0)
CPU is in 16-bit real mode and firmware like SeaBIOS takes over
SeaBIOS looks at first sector (512 bytes) of a bootable device
if the sector contains 0xAA55 at 510-byte offset, it's loaded at 0x7C00
load the rest of data manually from our boot device (e.g. on 0x8000)
jump to the code section we just loaded (e.g. to address 0x8000)
if necessary, set up Stack Pointer and Stack Segment

Getting to protected mode

make sure interrupts are disabled
set up and load a minimal GDT (Global Descriptor Table)
set PE (Protected Mode Enable) bit in CR0 (Control Register)
perform far jump selecting a proper GDT entry

GDT example

A minimal GDT code set up looks like the following:

gdt_start:
dq 0
gdt_code:; index 1
dw 0xFFFF; segment limit (low)
dw 0x0000; segment base (low)
db 0x00; base address (middle)
db 10011010b; access bits (read/write, executable, code or data, present)
db 11001111b; flags (32-bit protected, 4k blocks) - segment limit (high)
db 0x00; base address (high)

gdt_data:; index 2
dw 0xFFFF; segment limit (low)
dw 0x0000; segment base (low)
db 0x00; base address (middle)
db 10010010b; access bits (read/write, code or data, present)
db 11001111b; flags (32-bit protected, 4k blocks) - segment limit (high)
db 0x00; base address (high)
gdt_end:

gdt_descriptor:
dw gdt_end - gdt_start - 1
dd gdt_start

Then we enter protected mode by executing:

cli
lgdt [gdt_descriptor]

mov eax, cr0
or eax, 1
mov cr0, eax

jmp 0x08:protected_mode_start ; 0x08 is 1 << 3 (select index 1 in GDT)

The far jump reloads CS using the descriptor selected by 0x08. The other
segment registers must be loaded with appropriate protected-mode selectors as
well. Each segment register has a visible selector and hidden descriptor-cache
state containing information such as the base, limit and access rights.

How PVH helps

If we are in a virtual machine, we don't really need to go through all of these
steps. We don't need to have firmware or BIOS to set up CPU state properly, we
can simply do this from the VM.

Here, we can manipulate the vCPU to our will, and therefore we can set segment
registers ourselves to valid values, skipping GDT setup. We do need to do some
administration to help Linux understand what's going on and this is exactly
where PVH comes in.

ELF PVH note

To determine whether an x86 Linux ELF image supports the PVH boot ABI, we can
inspect its Xen ELF notes and look for XEN_ELFNOTE_PHYS32_ENTRY (note type
18). Its descriptor gives the physical 32-bit entry point at which the guest
should be started.

Newer kernels may also contain XEN_ELFNOTE_PHYS32_RELOC (type 19), which
describes alignment and physical-address constraints for relocatable PVH
kernels.

PVH boot memory setup

Boot procedure from PVH is more straightforward, but requires more memory
management from VM:

get RIP from ELF PVH note entry
load Linux kernel ELF segments into memory
copy kernel command line to memory
construct an E820-like memory map using HvmMemEntry entries
construct and fill out HvmStartInfo struct
load memory entries and HvmStartInfo to memory

If using initramfs you also need to load it into memory, along with its
separate initramfs command line. We also need to fill out HvmModuleEntry
struct to point to initramfs and its command line address. Note that
here we are not talking about kernel modules, but rather Xen modules, and
for Linux that only means initramfs.

A good reference for the structures above is actually in the Linux kernel
headers and can be found on:

https://elixir.bootlin.com/linux/v7.1.3/source/include/xen/interface/hvm/start_info.h

Setting up registers

RIP points to entrypoint from ELF note 18 entry
RBX points to HvmStartInfo physical address (this is PVH specific)
set up segment registers to valid Protected Mode state
enable PE bit in CR0 (as in ordinary boot)
initialize CR4 (Control Register 4) to zero
clear VM (bit 17), IF (bit 9) and TF (bit 8) in RFLAGS

Linux kernel-side PVH register setup is described here:

https://elixir.bootlin.com/linux/v7.1.3/source/arch/x86/platform/pvh/head.S#L30

Segment registers

I think the best source for how to set up segment registers is actually my own
code from my Rust KVM project:

let code = kvm_segment {
base: 0,
limit: 0xffff_ffff,
selector: 1 << 3,
type_: 0b1010, // 3rd bit: code/data = 1
   // 2nd bit: conforming = 0
   // 1st bit: readable = 1
   // 0th bit: accessed = 0
present: 1,
dpl: 0, // descriptor privilege level (0: kernel .. 3: userspace)
db: 1,  // default operand / address size (0: 16bit, 1: 32bit)
s: 1,   // system (1: code/data segment, 0: system segment)
l: 0,   // long mode (0: not 64bit segment, 1: 64bit segment)
g: 1,   // granularity (0: limit in bytes, 1: limit in 4Kb blocks)
avl: 0, // available for software (reserved for custom OS purposes)
unusable: 0,
padding: 0,
};

let data = kvm_segment {
selector: 2 << 3,
type_: 0b0010, // 3rd bit: code/data = 0
   // 2nd bit: conforming = 0
   // 1st bit: writable = 1
   // 0th bit: accessed = 0
..code };

sregs2.tr = kvm_segment {
base: 0,
limit: 0x67, // as per Xen project PVH protocol
g: 0,
..code };

With BIOS boot, these segment descriptors would live in a GDT in guest memory
and the CPU would obtain their hidden descriptor state by loading the
corresponding selectors. With KVM, we can bypass the GDT entirely and directly
initialize the vCPU's segment state through KVM_SET_SREGS.

Conclusion

PVH turned out to be a convenient way of getting my x86 KVM implementation to
the point where it could boot a real Linux kernel without first implementing
the traditional x86 boot protocol.

There is still plenty left to implement around the x86 side of the VMM, but
being able to reach the Linux kernel through PVH gives me a much more useful
starting point than reproducing the complete BIOS boot sequence first.

The implementation is available in the x86 initialization code in my kvm-rust
repository:

https://github.com/StjepanPoljak/kvm-rust/blob/main/src/arch/x86_64/init.rs

I plan to continue extending functionality of the x86 side of my VMM.

Booting ARM64 Linux in my Rust KVM Hypervisor: Getting to the console with help of eBPF

Fri, 25 Sep 2026 14:40:06 +0000

Recap

In the previous article I got ARM64 Linux as far
as executing inside my Rust KVM hypervisor. I still couldn't actually interact
with the guest, though: I had no working console.

This article is about getting from "the PC is moving" to typing "hello" into
Linux and getting "HELLO" back.

UART

Last time we learned how to obtain the device tree blob from QEMU's virt
machine. Now we want to take a look at what's inside. We can decompile it:

dtc -I dtb -O dts -o virt.dts virt.dtb

Then, a simple search through the file already gives us the clue we need:

pi@raspberrypi:~ $ grep -i uart virt.dts -A5 -B1
        pl011@9000000 {
                clock-names = "uartclk\0apb_pclk";
                clocks = <0x8000 0x8000>;
                interrupts = <0x00 0x01 0x04>;
                reg = <0x00 0x9000000 0x00 0x1000>;
                compatible = "arm,pl011\0arm,primecell";
};

So, now I had a lead: I needed to emulate the pl011 UART at 0x09000000 so
that the existing Linux pl011 driver could communicate with it. This was not a
really interesting part, as most of it was simply reading Linux kernel source
code along with pl011 documentation from ARM. Soon I was able to get output
from Linux and it felt really great:

[    0.000000] Linux version 7.1.3 (docker@stjepan-Aspire-F5-573G) (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT Thu Jul 16 08:52:24 UTC 2026
[    0.000000] KASLR disabled on command line
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: UEFI not found.
[    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x0000000047ffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x47fc10c0-0x47fc37bf]
                  ### omitted further kernel log ###

Creating custom initramfs

Now I needed bidirectional UART support so that I could send input to the guest
as well. To make further investigation easier, I created a very simple binary
written in C to be run as the Linux init process: it just prints out current
interrupt statistics and asks user for input, echoing it back in all caps. All
we then need is initramfs.txt file with some simple folders and devices:

dir / 0755 0 0
dir /dev 0755 0 0
dir /proc 0555 0 0
dir /sys 0555 0 0
nod /dev/kmsg 0600 0 0 c 1 11
nod /dev/console 0600 0 0 c 5 1
file /init /home/stjepanp/Develop/initramfs/init 0755 0 0

I only needed enough of a filesystem to provide /init, /dev/console, and
access to /proc for interrupt statistics. Then, in the Linux source code
folder, we generate initramfs with:

./usr/gen_initramfs.sh ../initramfs/initramfs.txt -o initramfs

To see at which memory address to add initramfs, we can run ordinary QEMU with
our initramfs and see what happens:

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt,dumpdtb=virt.dtb                                \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0 nokaslr rdinit=/init"          \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults                                             \
        -initrd "./initramfs"

After decompiling virt.dtb (explained in previous article) we can see the location:

pi@raspberrypi:~/dtb $ grep initrd virt-initramfs.dts -B2 -A3
        chosen {
                bootargs = "console=ttyAMA0 initramfs_async=0 rdinit=/init nokaslr";
                linux,initrd-start = <0x0 0x40200000>;
                linux,initrd-end = <0x0 0x40294200>;
                stdout-path = "/pl011@9000000";
                kaslr-seed = <0x8114b902 0xe815aa93>;
        };

We do have to recalculate initrd-end if size of our initramfs file changes.
If that happens, we can easily recompile the device tree blob after changing
the value. Also, let's not forget to load initramfs in Rust code:

const LOAD_ADDR: u64 = 0x40_000_000;
const INITRAMFS_OFFS: u64 = 0x200_000;
const KERN_OFFS: u64 = 0x200_000 + INITRAMFS_OFFS;

impl VM {
    pub fn arch_load_linux(&mut self, vcpu: &mut VCPU, args: &Args) -> io::Result<()> {
        let linux_mem_idx = self.add_mem_region(1024 * 1024 * 1024, LOAD_ADDR)?;
        self.load_file_to_memory(linux_mem_idx, &dtb_path, 0x0)?;
        self.load_file_to_memory(linux_mem_idx, &initramfs_path, INITRAMFS_OFFS)?;
// ...
    }

Setting up vGIC

Even without having the ability to send keystrokes there was one big problem
with the Linux console stdout: The kernel messages were reaching the console,
but once execution reached /init, output stopped behaving as expected. It
took me some time to realize that the ARM architectural timer was also involved.
Its interrupts need to be delivered through the virtual GIC, so I first needed
to create a vGIC. It turned out creating the vGIC was easy. I just followed the
logic from QEMU source code:

https://elixir.bootlin.com/qemu/v11.0.1/source/accel/kvm/kvm-all.c#L4035

In Rust this translates to:

let mut create_device: kvm_create_device = unsafe { std::mem::zeroed() };
create_device.type_ = KVM_ARM_DEVICE_VGIC_V2;
create_device.fd = 0xffffffff;
create_device.flags = 0;

unsafe { libc::ioctl(self.fd, KVM_CREATE_DEVICE, &mut create_device); }

This worked. Also note that I have omitted proper error handling to keep the
code short and concise.

Tracing IRQ injection

Now if we run our initramfs with strace as before, and we try to type in
some keys, we will see that in our log we have:

pi@raspberrypi:~ $ grep -i irq kvm.log | tail -n5
458398 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7fd932dbd0) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db30) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db20) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0

So our ioctl for IRQ injection actually takes an address as argument. If we
investigate this, we can find out how KVM_IRQ_LINE is handled in the Linux
kernel:

https://elixir.bootlin.com/linux/v6.1.21/source/virt/kvm/kvm_main.c#L4776

Reading this code we see that the relevant function is kvm_vm_ioctl_irq_line
that takes struct kvm_irq_level as second argument and this is what we need
to interpret:

/* for KVM_IRQ_LINE */
struct kvm_irq_level {
/*
 * ACPI gsi notion of irq.
 * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47..
 * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23..
 * For ARM: See Documentation/virt/kvm/api.rst
 */
union {
__u32 irq;
__s32 status;
};
__u32 level;
};

For this purpose, we can treat it as an 8-byte structure containing the IRQ number and level. Therefore, we can easily inspect it again via eBPF when we
run QEMU with our initramfs:

pi@raspberrypi:~ $ sudo bpftrace -e 'kprobe:kvm_vm_ioctl_irq_line { printf("irq_level=%x\n", *((uint64 *)arg1)); }'
Attaching 1 probe...
irq_level=1000021
irq_level=1000021
irq_level=1000021
irq_level=1000021

Without wasting much time we can easily reconstruct this behavior in Rust:

pub fn arch_update_irq(level: u32, vm_fd: libc::c_int) -> io::Result<()> {
    let mut irq: kvm_irq_level = unsafe { std::mem::zeroed() };
    irq.level = level;
    irq.__bindgen_anon_1.irq = 0x1000021;

    let ret = unsafe {
        libc::ioctl(vm_fd, KVM_IRQ_LINE, &mut irq)
    };

    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    Ok(())
}

Note that, in essence, 0x1000021 is not an arbitrary magic number. On ARM64,
KVM_IRQ_LINE encodes the interrupt type and interrupt ID in the irq field. Here 0x01 selects an SPI and 0x21 is interrupt 33 the UART interrupt from
the device tree.

No such device or address

However, when we run our program, it fails with "No such device or address". To
get some clue on what is going on, we can try strace and get:

ioctl(5, KVM_RUN, 0)                    = -1 ENXIO (No such device or address)
Error: Os { code: 6, kind: Uncategorized, message: "No such device or address" }

My first impulse was to search for ENXIO in Elixir bootlin, but,
unfortunately, there were simply too many functions and branches that could
return ENXIO. Rather than trying to follow every possible ENXIO path
manually, I decided to use a more brute-force approach. I attached kretprobes
to the KVM functions and looked for one returning -ENXIO:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:kvm_* /retval == (uint32)(-6)/ { printf("%s\n", probe); }'
Attaching 487 probes...
kretprobe:kvm_vgic_map_resources
kretprobe:kvm_arch_vcpu_run_pid_change

The second function here does not seem really relevant, so I tried looking for
kvm_vgic_map_resources on Elixir:

https://elixir.bootlin.com/linux/v6.1.21/source/arch/arm64/kvm/vgic/vgic-v2.c#L289

Now taking a look at the function we can clearly see where ENXIO is returned:

int vgic_v2_map_resources(struct kvm *kvm)
{
    struct vgic_dist *dist = &kvm->arch.vgic;
    int ret = 0;

    if (IS_VGIC_ADDR_UNDEF(dist->vgic_dist_base) ||
        IS_VGIC_ADDR_UNDEF(dist->vgic_cpu_base)) {
        kvm_debug("Need to set vgic cpu and dist addresses first\n");
        return -ENXIO;
    }
    // ...

Now we have a hint. We need to set vGIC CPU and distributor addresses first. To
confirm our hypothesis we can try tracing this function specifically:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:vgic_v2_map_resources { printf("ret=%u\n", retval); }'
Attaching 1 probe...
ret=4294967290

Finding out distributor address

My first attempt to set up distributor address didn't go well. I wrote the
following code (I extracted distributor address from the device tree and
tried to reconstruct the logic from QEMU source code):

let dist_addr: u64 = 0x08_000_000;
let mut dev_attr: kvm_device_attr = unsafe { std::mem::zeroed() };
dev_attr.group = KVM_DEV_ARM_VGIC_GRP_ADDR;
dev_attr.attr = KVM_VGIC_V2_ADDR_TYPE_DIST;
dev_attr.flags = 0;
dev_attr.addr = dist_addr;

unsafe { libc::ioctl(create_device.fd as i32, KVM_SET_DEVICE_ATTR, &mut dev_attr) };

However, when I ran the program, I got:

ioctl(6, KVM_SET_DEVICE_ATTR, 0x7ffe2696d8) = -1 EFAULT (Bad address)

To debug this, I needed to inspect the values defined in
struct kvm_device_attr, but it was a lot more complicated: I had to check if
the struct I was tracing was for KVM_DEV_ARM_VGIC_GRP_ADDR group and for
KVM_VGIC_V2_ADDR_TYPE_DIST. So, I decided to write my eBPF program in C with
and called it probe-kvm.bpf.c:

#include <linux/bpf.h>
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <linux/kvm.h>
#include <stdbool.h>
#include <stdint.h>

char LICENSE[] SEC("license") = "GPL";

SEC("kprobe/vgic_v2_set_attr")
int BPF_KPROBE(kprobe_vgic_v2_set_attr, void *dev, struct kvm_device_attr *dev_attr)
{
        struct kvm_device_attr attr;
        __u64 value;

        bpf_probe_read_kernel(&attr, sizeof(attr), dev_attr);
        if (attr.group != KVM_DEV_ARM_VGIC_GRP_ADDR) {
                return 0;
        }

        if ((attr.attr != KVM_VGIC_V2_ADDR_TYPE_DIST) && (attr.attr != KVM_VGIC_V2_ADDR_TYPE_CPU)) {
                return 0;
        }

        bpf_printk("\nset_attr: attr=%s addr=0x%llx", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr);

        return 0;
}

To compile and run it I wrote the following script:

#!/bin/sh

clang -O2 -g -mcpu=v1 -target bpf -I /usr/include/aarch64-linux-gnu -c probe-kvm.bpf.c -D__TARGET_ARCH_arm64 -o probe-kvm.bpf.o
sudo rm -rf /sys/fs/bpf/vgic
sudo bpftool prog loadall probe-kvm.bpf.o /sys/fs/bpf/vgic autoattach
sudo cat /sys/kernel/debug/tracing/trace_pipe

So, I started ordinary QEMU along my eBPF trace and got this output:

$ ./start-probe.sh
           <...>-146994  [002] d..31 953951.139387: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fd4d3c450

           <...>-146994  [002] d..31 953951.139418: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fd4d3c450

This really looked like a userspace address while I was supplying a raw address.
To find out what was at that address I modified the last line of the BPF program
above:

        if (bpf_probe_read_user(&value, sizeof(value), (void *)attr.addr) != 0)
                return -1;

        bpf_printk("\nset_attr: attr=%s addr=0x%llx, *addr=0x%llx\n", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr, value);

With this I got what I didn't really expect:

           <...>-147143  [000] d..31 954397.183145: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fdc806ca0, *addr=0x8010000

           <...>-147143  [000] d..31 954397.183177: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fdc806ca0, *addr=0x8000000

So the addr field is actually a pointer to the address, and not the address
itself. Note that 0x8010000 and 0x8000000 are the guest physical
addresses of the GIC CPU interface and distributor taken from the virt machine's
device tree:

pi@raspberrypi:~/dtb $ grep -i intc virt.dts -A3
        intc@8000000 {
                phandle = <0x8001>;
                reg = <0x00 0x8000000 0x00 0x10000 0x00 0x8010000 0x00 0x10000>;
                compatible = "arm,cortex-a15-gic";

This also explained the EFAULT: KVM was trying to dereference
attr.addr as a userspace pointer. With this small fix, I got a fully working
Linux console:

dev_attr.addr = &dist_addr as *const u64 as u64;

The following snippet shows that my init process (as part of
initramfs) was run and it echoed back my input in capital letters:

[    0.832623] clk: Disabling unused clocks
[    0.834046] PM: genpd: Disabling unused power domains
[    0.845565] Freeing unused kernel memory: 12672K
[    0.847475] Run /init as init process
           CPU0       
 11:        199 GIC-0  27 Level     arch_timer
 13:          0 GIC-0  33 Level     uart-pl011
IPI0:         0       Rescheduling interrupts
IPI1:         0       Function call interrupts
IPI2:         0       CPU stop interrupts
IPI3:         0       CPU stop NMIs
IPI4:         0       Timer broadcast interrupts
IPI5:       168       IRQ work interrupts
IPI6:         0       CPU backtrace interrupts
IPI7:         0       KGDB roundup interrupts
Err:          0

type a string: hello
allcaps: HELLO

[    2.224776] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    2.227382] CPU: 0 UID: 0 PID: 1 Comm: init Not tainted 7.1.3 #1 PREEMPT 
[    2.229749] Hardware name: linux,dummy-virt (DT)

The interrupt statistics are also printed from my init. Also, when init
exits, we get the (correct) Kernel panic that init process was terminated.

Final thoughts

At this point I finally had what I was looking for: a real Linux userspace
running inside my own KVM-based hypervisor, with a working UART, interrupt
delivery and a minimal initramfs.

More importantly, I now had a way to interact with the guest and inspect what
was happening inside KVM. That gives me a much better foundation for the next
step: implementing more of the virtual hardware needed to turn this into a
useful ARM64 virtual machine.

You can also check out the repository on my GitHub page and, if you have a
Raspberry Pi 4B, try the VMM yourself:

https://github.com/StjepanPoljak/kvm-rust

What is GPU Passthrough?

Mon, 21 Sep 2026 14:40:10 +0000

GPU passthrough is a virtualisation method used to directly assign a physical graphics processing unit to a single VM, ultimately bypassing the virtualisation software layers like the NVIDIA Virtual GPU (vGPU) manager. In this configuration, the physical hardware is accessed exclusively by the driver running inside that specific VM and is not shared with other environments. While a single VM can be assigned multiple physical GPUs depending on hypervisor limits, individual physical GPUs cannot simultaneously host virtual GPUs and perform passthrough.

This architectural limitation creates challenges for local development setups. For instance, I was trying to run a simple Kubernetes cluster through Docker Desktop on my Apple Silicon Macbook and I had encountered a now obvious hardware limitation because macOS container runtimes operates inside a Linux VM via the Hypervisor framework. This setup cannot natively pass through the Apple Silicon GPU using Metal to containers. As a result, running even a simple, small qwen2.5:7b model relied entirely on CPU software inference, which easily maxed-out the processor cores and caused noticeable performance bottlenecks. Like, I was unable to perform even the most basic tasks with this local model.

To address hardware isolation and GPU utilisation constraints in production environments, tools like Edera allow direct NVIDIA and AMD GPU passthrough into isolated Edera zones. By configuring hardware identifiers in the protect daemon, binding devices to the appropriate drivers, and launching zones with dedicated kernel variants, users can run GPU-accelerated workloads and verify hardware access directly through utilities like nvidia-smi.

Bryan Steiner has a really useful tutorial for GPU passthrough, if this is interesting to readers:
https://github.com/bryansteiner/gpu-passthrough-tutorial

Booting ARM64 Linux in my Rust KVM Hypervisor: Debugging a Memory Issue

Sun, 06 Sep 2026 18:39:35 +0000

Motivation

The time has come: the big test. I got some simple binaries running on my Rust
KVM hypervisor (at this point I am thinking about giving it a proper name). The
next thing has to be Linux. For this series I will show how to actually boot
Linux on my Rust KVM hypervisor on ARM64 CPU (I am using Raspberry Pi 4 which
supports hardware accelerated virtualization).

Cross-compiling Linux kernel

In one of my previous articles I have described how to
set up the cross-compilation environment with a simple Dockerfile. We can
reuse it to cross-compile the Linux kernel for ARM64 (after adding some Linux
build dependencies like bc, flex, python3, libssl-dev and bison):

docker run --network host -it --rm \
       -e CROSS_COMPILE=aarch64-linux-gnu- -e ARCH=arm64 \
       -v "$(pwd)":"/home/docker/kvm-rust" -w "/home/docker/kvm-rust/linux-7.1.3" \
       arm-rust-build /bin/bash -c 'make defconfig && make'

The resulting ARM64 Linux kernel image (Image) should be located in
arch/arm64/boot/ directory.

ARM64 boot protocol

The full ARM64 Linux boot protocol is quite straightforward and can be found in
the Linux kernel documentation:

https://www.kernel.org/doc/html/v7.1/arch/arm64/booting.html

We simply need to parse the 64-byte header and verify the magic field (it should
equal 0x644d5241). I will just highlight a few important points. We should
read image_size and text_offset fields and note that the image should be
loaded at an offset of text_offset bytes at a 2 MiB boundary. For our kernel,
as I checked, text_offset is zero and image_size is the amount of memory the
decompressed kernel image occupies in RAM, which is not necessarily the same as
the file size as reported by stat. Finally, the registers x1, x2 and x3
must be set to zero, while x0 must point to the device tree blob (dtb) in
the RAM. So, for this I chose to load the dtb at address 0x40000000 and
Image itself on the first 2 MiB boundary:

const LOAD_ADDR: u64 = 0x40_000_000;
const KERN_OFFS: u64 = 0x200_000;

Obtaining the device tree blob

So, for ARM64 we do need a device tree blob that will tell the Linux kernel on
what kind of hardware it is running on. And this step is quite easy, we will
can simply reuse our command for running QEMU, with one small addition (i.e.
adding dumpdtb option to -M virt argument):

#!/bin/sh

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt,dumpdtb=virt.dtb                                \
        -smp 1                                                  \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0"                               \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults

This gives us a consistent snapshot of the hardware we were running on (in this
case it's emulated hardware for QEMU virt machine). We could (and we should)
edit and prune it for our own purposes, but let's leave that for another time.

Implementing boot in Rust

So, loading the kernel in Rust is quite straightforward:

impl VM {
    pub fn arch_load_linux(&mut self, vcpu: &mut VCPU, args: &Args) -> io::Result<()> {
        let header = get_linux_header(&args.binary)?;
        let linux_mem_idx = self.add_mem_region((KERN_OFFS + header.image_size) as usize, LOAD_ADDR)?;

        match &args.dtb {
            Some(dtb_path) => { self.load_file_to_memory(linux_mem_idx, &dtb_path, 0x0)?; },
            None => { return Err(io::Error::other("Device tree blob not specified.")); }
        }

        self.load_file_to_memory(linux_mem_idx, &args.binary, KERN_OFFS)?;

        vcpu.set_one_reg("x0", LOAD_ADDR)?;
        vcpu.set_one_reg("x1", 0x0)?;
        vcpu.set_one_reg("x2", 0x0)?;
        vcpu.set_one_reg("x3", 0x0)?;

        vcpu.set_one_reg("pc", LOAD_ADDR + KERN_OFFS)?;
    }
}

Note that the get_linux_header() function was omitted here as it only parses
the aforementioned ARM64 Linux image headers in a boilerplate manner.

Function not implemented

As soon as I tried to run my hypervisor with Linux Image, I ran into an issue:

pi@raspberrypi:~ $ ./rust -d virt Image 
x0 = 0x0
pc = 0x1000
magic 0x644d5241, image_size: 38731776, file_size: 38040064, text_offset: 0x0
Loading "virt" to 0x0...
Loading "Image" to 0x200000...
x0 = 0x40000000
pc = 0x40200000
x0 = 0xfffffbfffda35000
pc = 0xffffd902f0f8a42c
Error: Os { code: 38, kind: Unsupported, message: "Function not implemented" }

Trying strace

First thing I turned to was strace; it's easy to use and gets some nice
information really fast:

ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee98768) = 0
x0 = 0x40000000
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee98768) = 0
pc = 0x40200000
ioctl(5, KVM_RUN, 0)                    = -1 ENOSYS (Function not implemented)
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee988a8) = 0
x0 = 0xfffffbfffda35000
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee988a8) = 0
pc = 0xffffd902f0f8a42c
Error: Os { code: 38, kind: Unsupported, message: "Function not implemented" }
+++ exited with 1 +++

I didn't learn much other than the fact that KVM_RUN failed completely with
ENOSYS error code. My first attempt to find out what was going on was
searching the ARM64 KVM source code for ENOSYS in Linux repository. It didn't
take long, as it appeared only in one place which was a promising lead:

https://elixir.bootlin.com/linux/v7.1.3/source/arch/arm64/kvm/mmio.c#L188

As we can see, the function in question is io_mem_abort() in the Linux kernel.
Now, strace got us that far, but this is a Linux kernel function and not much
we can find out more about it with strace (which traces only system calls like
ioctl). I haven't even considered GDB because this was the host kernel,
and this would require a really complex time-consuming setup. Luckily, there was
a much easier way.

Analyzing IO memory abort with eBPF

The first thing I did was to list kprobes in bpftrace and see if I can hook
into the io_mem_abort at all via eBPF:

pi@raspberrypi:~ $ sudo bpftrace -l 'kprobe:io_mem_abort'
kprobe:io_mem_abort

I was in luck, so I could at least confirm that this function was the one
causing the issue, so I attached a kretprobe to it, ran bpftrace alongside
my hypervisor and got the confirmation I needed:

pi@raspberrypi:~ $ sudo bpftrace -e '
>    kretprobe:io_mem_abort {
>        printf("ret=%d\n", retval);
>    }'
Attaching 1 probe...
ret=-38

Finding out what really happened

Now that I have confirmed io_mem_abort was the culprit, it was time to
analyze it more deeply. I attached an additional kprobe so I could print both
the return value and the fault_ipa argument:

int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)

The updated bpftrace script printed the fault address.

pi@raspberrypi:~ $ sudo bpftrace -e '
>    kretprobe:io_mem_abort {
>        printf("ret=%d\n", retval);
>    }
>    kprobe:io_mem_abort {
>        printf("0x%llx\n", arg1);
>    }'
Attaching 2 probes...
0x47fff000
ret=-38

So the result surprised me as I initially suspected it was trying to access some
device that I haven't created; however, I realized that this was actually just
beyond the memory I had allocated for the Linux kernel to use. So it clicked: I probably hadn't allocated enough guest memory. Instead of allocating just
header.image_size + KERN_OFFS I increased it to 1 GiB:

let linux_mem_idx = self.add_mem_region(1024 * 1024 * 1024, LOAD_ADDR)?;

Conclusion

With that adjustment, the ENOSYS error disappeared. The guest was clearly
executing code, although I still wasn't seeing any output on the serial console.
That meant the memory issue was solved, but there was at least one more thing
left before we would actually see Linux boot.

KVM vs KVM - what is the question?

Tue, 28 Jul 2026 22:04:05 +0000

Last weekend I was looking for a new project for my Raspberry Pi. I was looking for ways to set up a KVM on my Pi5 since it has 8GB of RAM, which is beefy enough for additional virtualisation tasks. Once I started Googling how I should start this task, I would look up “KVM on Raspberry Pi”. Most of the search results were for another entirely different KVM.

A Keyboard, Video, and Mouse (KVM) switch is a hardware device that allows a user to control multiple computers or servers using a single set of keyboard, video display, and mouse devices. While the KVM switch is a great technology that allows you to control multiple computers using your existing setup, I currently use Tailscale for accessing my remote Pis via my home laptop. So that wasn’t what I was looking for. Instead, I was interested in seeing whether I could run a Kernel Virtual Machine (KVM) on my Pi5.

Despite sharing the same acronym, a KVM switch and the KVM software operate in entirely different realms of technology: one is a physical hardware device for managing physical computers, while the other is open-source software built into Linux to create and manage virtual machines. The KVM Switch routes peripheral and display signals between different systems with the press of a button or a keyboard shortcut.

What is a KVM VM?

Kernel-based VM is an open source virtualisation technology for Linux. With KVM, Linux can function as a hypervisor that runs multiple, isolated VMs. KVM was announced way back in 2006 and actually merged into the Linux kernel in 2007. Virtualisation, which lets you run an operating OS inside another OS, is possible thanks to hypervisors.

A hypervisor pools computing resources (like processing, memory, and storage) and reallocates them among VMs. The hypervisor can run multiple VMs at once, manage them, and support the creation of new ones. The physical hardware, when used as a hypervisor, is called the host, while the many VMs that use its resources are guests. Hypervisors need OS-level components such as memory management, process scheduling, I/O, device drivers, security management, a network stack, and more to run VMs.

KVM has all these components because it’s part of the Linux kernel. Every VM is implemented as a regular Linux process, scheduled by the standard Linux scheduler, with dedicated virtual hardware like a network card, graphics adapter, CPU, memory, and disks.

Similarities between KVM and KVM

Both technologies fundamentally exist to solve an efficiency problem: enabling a user to manage multiple OSs from a single workstation without buying duplicate input devices or displays. Whether toggling between two physical tower PCs or switching focus between multiple virtual instances, both implementations allow one operator to interact with distinct, isolated computing environments. Furthermore, both solutions preserve system isolation, ensuring that actions taken within one environment do not interfere with or crash the other operating systems running alongside it.

Differences

The key distinction lies in hardware physical routing versus software virtualisation. A KVM switch is a tangible piece of hardware that physically redirects signal cables (HDMI/DisplayPort and USB) so that a single set of peripherals can control multiple physically distinct servers or PCs. It does not alter how the computers operate; it merely changes which machine receives your keypresses and sends video output to your monitor.

In contrast, a Kernel-based VM is a software module integrated directly into the Linux kernel that turns the host OS into a Type-1 hypervisor. Instead of connecting separate physical machines, the kernel-based KVM partitions a single computer's underlying processor, memory, and storage to simulate entirely separate virtual computers (Guest VMs) on the same bare metal.

The question should’ve been KVM or Xen?

And that brings us to a totally different question. KVM and Xen - what are the differences and which one can I run on my Pi5 for virtualisation? While both KVM and Xen are open-source Type-1 hypervisors that deliver high-performance virtualisation, their core designs and operational structures differ significantly.

KVM is built directly into the Linux kernel as a standard kernel module (kvm.ko). Installing KVM turns an existing Linux OS into a hypervisor, granting you immediate access to standard Linux drivers, process schedulers, and existing system management tools without having to modify the host kernel. Conversely, Xen sits below the OS as a microkernel-based layer that boots first, before any OS. It relies on a special, privileged Guest VM known as Dom0 (Domain 0) to handle hardware access, boot management, and admin control over other Guest VMs (DomU).

KVM relies purely on hardware-assisted virtualisation (using processor extensions like Intel VT-x or ARM Virtualisation Extensions) to run unmodified Guest OSs. Xen historically pioneered paravirtualisation (where guest kernels are modified to communicate directly with the hypervisor API for higher efficiency), though it now supports hardware-assisted virtualisation as well. Furthermore, Xen’s strict separation of the hypervisor microkernel from the control domain (Dom0) isolates hardware drivers away from guest VMs, which makes it a popular architecture for high-security environments like Qubes OS.

For almost all Raspberry Pi 5 use cases, KVM is the far more practical and seamless option.

In the case of the Raspberry Pi, you’ll need to verify hardware virtualisation support before doing anything. Check that ARM CPU virtualisation extensions are available. Check that the KVM hypervisor kernel module is present and accessible with the below command:

ls -l /dev/kvm

You should see an output showing crw-rw---- 1 root kvm ... /dev/kvm.
If /dev/kvm exists, hardware acceleration is supported and active.

The Raspberry Pi’s boot process relies on a proprietary EEPROM-based bootloader that loads GPU firmware and kernel binaries directly from an SD card or NVMe storage (often chainloaded through U-Boot for standard Linux environments). Because KVM is integrated into standard ARM64 Linux kernels (like Raspberry Pi OS 64-bit or Ubuntu), it requires no special bootloader tweaks or UEFI emulation layer. You simply install `qemu-kvm` (QEMU) and libvirt, and the Pi instantly acts as a hypervisor using built-in ARM hardware extensions, allowing me to easily spin up VMs using standard Linux tools like virt-manager or cockpit.

You can run the following command to install the hypervisor, daemon services, and CLI management tools like QEMU core binaries. Since I’m running Raspberry Pi OS Desktop with the graphical interface, I also appended virt-manager to the below command:

sudo apt install -y qemu-system-arm qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager

Once the packages are installed, I just need to startup/enable the libvirtd service to ensure the hypervisor daemon starts automatically on boot:

sudo systemctl enable --now libvirtd

Libvirt includes a default NAT network interface (virbr0) for guest VMs. As soon as the default virtual network was configured, I can simply create my first Guest VM on the Pi5 and use an Ubuntu ARM64 ISO image, sourced from the Canonical website. The virt-install looks a little like this:

virt-install \
  --name debian-vm \
  --memory 2048 \
  --vcpus 2 \
  --disk size=10,bus=virtio \
  --os-variant debian12 \
  --cdrom /path/to/debian-arm64.iso \
  --network network=default \
  --graphics vnc,listen=0.0.0.0

In contrast, setting up Xen on a Pi 5 requires a complex boot architecture. Because Xen acts as a microkernel that must load before any OS, booting it on an SBC (Single Board Computer) generally requires using U-Boot to load the Xen binary alongside device-tree overlays, a customised Dom0 control domain, and a secondary driver domain. While projects like Xen-Troops exist specifically to compile custom Yocto-based Xen images for embedded systems and automotive apps, setting this up for home or homelab experiments involves significant compilation overhead, non-standard driver configs, and manual partition setups. While maybe not suited to Pi5, it’s not impossible on previous versions like the Pi4, as stated on the Linux blog. For edge use-cases you can certainly stick with KVM for a smooth setup unless you specifically need Xen's compartmentalised driver architecture for complex embedded development. If you’d like to read why Edera specifically opted for Xen when building a secure microservice foundation, check out their blog post.

Porting my Rust KVM hypervisor to ARM64: Running the binary

Wed, 22 Jul 2026 18:02:25 +0000

Recap

In the previous article we ported the register handling
of my Rust KVM hypervisor from x86 to ARM64. By tracing QEMU with strace and
inspecting the Linux headers, we discovered that ARM64 accesses CPU state
through the KVM_GET_ONE_REG and KVM_SET_ONE_REG interface and reconstructed
the register id encoding in Rust.

At that point the hypervisor compiled successfully, but trying to access even
the program counter resulted in an ENOEXEC error. In this article we'll
continue the reverse-engineering process, initialize the vCPU correctly, and
finally execute our first ARM64 guest.

Second roadblock: ENOEXEC when getting / setting PC

I was rather optimistic this would work out of the box, but even after trying
to do something as simple as reading the PC register I hit a snag:

ioctl(5<anon_inode:kvm-vcpu:0>, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fdf2c81e8) = -1 ENOEXEC (Exec format error)
Error: Os { code: 8, kind: Uncategorized, message: "Exec format error" }

Luckily, I had a concrete lead now: error code and KVM command. So I searched
for the KVM_GET_ONE_REG in the Linux kernel code on Bootlin Elixir and found
the following code in arch/arm64/kvm/arm.c:

case KVM_SET_ONE_REG:
case KVM_GET_ONE_REG: {
struct kvm_one_reg reg;

r = -ENOEXEC;
if (unlikely(!kvm_vcpu_initialized(vcpu)))
break;

This is exactly the error we were getting and we immediately found what was
missing: vCPU initialization. Actually, if we take a closer look at the code we
will see the comment kernel developers have left us:

int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
{
int err;

/* Force users to call KVM_ARM_VCPU_INIT */
vcpu->arch.target = -1;

Initializing the vCPU

Following the lead I looked for KVM_ARM_VCPU_INIT in the original strace log
and found:

137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57e028) = 0
137156 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7f82f70e68) = 0
137152 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57dd78) = 0
137152 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57e048) = 0

This wasn't really helpful: we again only see addresses, but we see no hints of
underlying structure. So let's first take a look at how KVM_ARM_VCPU_INIT is
defined in the Linux headers:

$ grep -Rn KVM_ARM_VCPU_INIT /usr/include/
/usr/include/linux/kvm.h:1622:#define KVM_ARM_VCPU_INIT   _IOW(KVMIO,  0xae, struct kvm_vcpu_init)

A quick search for struct kvm_vcpu_init revealed:

$ grep -Rn 'struct kvm_vcpu_init' /usr/include/ -A3
/usr/include/aarch64-linux-gnu/asm/kvm.h:112:struct kvm_vcpu_init {
/usr/include/aarch64-linux-gnu/asm/kvm.h-113-   __u32 target;
/usr/include/aarch64-linux-gnu/asm/kvm.h-114-   __u32 features[7];
/usr/include/aarch64-linux-gnu/asm/kvm.h-115-};

Getting struct data with Python GDB

I didn't want to spend much time understanding every target and feature flag. My
goal was simply to reproduce QEMU's behavior as quickly as possible (and I
didn't even have to understand it completely). Luckily, this structure was
really simple (basically an array of eight 32-bit integers), so I decided to
tackle this with Python GDB automation. I created a breakpoint to stop exactly
on ioctl call when we are in KVM_ARM_VCPU_INIT:

import gdb
import struct

def log(line):
    print(line)
    with open("gdb-debug.out", "a") as f:
        f.write(line + "\n")

class IoctlBreakpoint(gdb.Breakpoint):

    def stop(self):
        if x1 == 0x4020aeae: # KVM_ARM_VCPU_INIT
            ptr = int(gdb.parse_and_eval("$x2"))
            vcpu_init_mem = gdb.selected_inferior().read_memory(ptr, 32) # u32 + 7 * u32
            target_features = struct.unpack("<8I", bytes(vcpu_init_mem))
            log(f"KVM_ARM_VCPU_INIT: target={target_features[0]} features={target_features[1:]}")        
            return False

if __name__ == "__main__":
    tracer = IoctlBreakpoint("ioctl")
    gdb.execute("handle SIGUSR1 SIGUSR2 nostop noprint pass")
    gdb.execute("run")

I ran my QEMU command with GDB:

gdb -ex 'source ./trace-arm.py' ./start-qemu.sh

This has automatically yielded great results:

KVM_ARM_VCPU_INIT: target=5 features=(0, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)

For now I want to try just setting target, but leave features zeroed out. I
could immediately reconstruct this in Rust:

impl VCPU {
    pub fn arm_vcpu_init(&self) -> io::Result<()> {
        let mut vcpu_init : kvm_vcpu_init = unsafe { std::mem::zeroed() };
        // KVM_ARM_VCPU_INIT: target=5 features=(0, 0, 0, 0, 0, 0, 0)
        vcpu_init.target = 5;
        let ret = unsafe { libc::ioctl(self.fd, KVM_ARM_VCPU_INIT, &mut vcpu_init) };
        if ret < 0 {
            return Err(io::Error::last_os_error());
        }
        Ok(())
    }
}

Third roadblock: No KVM exit

Soon after this I was able to get and set registers properly. The first ARM test
binary I compiled was based on working UART code from my hobby operating system
I started on Raspberry Pi. I expected at least a fault exit, if not an MMIO exit
when UART code was called. However, KVM_RUN never returned.

This is where I was stuck for a while. I considered creating a separate thread
that repeatedly queried KVM_GET_ONE_REG for PC, but from my own working
experience I knew that calling KVM ioctl calls asynchronously was a really bad
idea. I also considered adding hardware breakpoint capabilities to at least
trigger KVM_EXIT_DEBUG, but this would require more infrastructure for just
checking simple binary.

Writing a simple binary

I decided to try adding a signal handler instead. If I interrupted KVM_RUN
call with SIGINT maybe I could get registers then synchronously. I lowered
my expectations for initial binary and just created something really simple:

.section .text

.macro curr_el_to reg
mrs\reg, CurrentEL
lsr\reg, \reg, #2
and\reg, \reg, #0xf
.endm

.globl _start
_start:
mrsx0, mpidr_el1
andx0, x0, #0xFF
cbzx0, control
b .

control:
curr_el_to x0
b .

It looks like much, but this code will just load current CPU privilege level to
x0 register (being that level EL=1 is operating-system level, x0 should be
set to 0x1). The mpidr_el1 code is for checking if we are on control CPU
(this should be unnecessary here, but I added this out of paranoia caused by
real-life issues I had on ARM).

Installing signal handler

Finally, I added a signal handler:

extern "C" fn handler(_sig: libc::c_int) { }

unsafe fn install_interrupt_signal() {
    let mut sa: libc::sigaction = std::mem::zeroed();
    sa.sa_sigaction = handler as *const() as usize;
    libc::sigemptyset(&mut sa.sa_mask);
    sa.sa_flags = 0;
    libc::sigaction(libc::SIGINT, &sa, std::ptr::null_mut());
}

The handler itself does nothing. I only needed the signal to interrupt the blocking KVM_RUN call. So my final KVM loop looked like:

    unsafe { install_interrupt_signal() };

    let run = vcpu.kvm_run_mem as *mut kvm_run;

    loop {
        let ret = unsafe { libc::ioctl(vcpu.fd, KVM_RUN, 0usize) };
        if ret < 0 {
            if io::Error::last_os_error().raw_os_error() == Some(libc::EINTR) {
                vcpu.print_regs()?;
            }
            return Err(io::Error::last_os_error());
        }

        let exit_reason = unsafe { (*run).exit_reason };

Running the binary

Finally, it was time to run the gues:

pi@raspberrypi:~ $ ./rust hello-world-arm.img
x0 = 0x0
pc = 0x1000
Loading "hello-world-arm.img" to 0x1000...
^Cx0 = 0x1
pc = 0x101c
Error: Os { code: 4, kind: Interrupted, message: "Interrupted system call" }

We can see that PC did change and that x0 really is set to current CPU
privilege level: 0x1. Why it didn't work for my UART code is probably that the
guest itself requires more initialization than I though (x86 went more smoothly
and felt more plug-and-play).

Conclusion

Although x86 and ARM64 share the same KVM interface at a high level, their
userspace APIs differ significantly. Porting my hypervisor turned out to involve
much more than renaming registers: ARM64 uses per-register ids instead of a
single register structure, requires explicit vCPU initialization through
KVM_ARM_VCPU_INIT and demanded a fair amount of reverse engineering before I
could successfully execute guest code. Fortunately, the same workflow that
worked on x86 like combining strace, GDB and the Linux kernel source proved just
as effective on ARM64 (although required a bit more effort).

The code from this article can be found on my GitHub page:

https://github.com/StjepanPoljak/kvm-rust/tree/kvm-arm-part2-code

Porting my Rust KVM hypervisor to ARM64: Working with registers

Fri, 10 Jul 2026 17:44:14 +0000

Introduction

This is a follow-up to my KVM in Rust series where I showed how to use strace
and GDB to reverse-engineer QEMU/KVM and reimplement pieces of it in my own
hypervisor in Rust. So far this was done on a modern x86 CPU. This time we'll
see how to get KVM running on 64-bit ARM architecture. For this I have used my
Raspberry Pi 4B which has a CPU with hardware virtualization support.

Cross-compilation

The first thing to do was to set up an environment for cross-compilation and for
this kind of thing I almost always use docker. So I have written a simple
Dockerfile installing the cross-compilation toolchain for 64-bit ARM along
with Rust essentials:

https://github.com/StjepanPoljak/kvm-rust/blob/kvm-arm-code/Dockerfile

Building this image is straightforward, but it does require passing user and
group id to have the resulting file permissions correct:

docker build --network host \
       --build-arg GID=$(id -g) \
       --build-arg=UID=$(id -u) \
       -t arm-rust-build .

Finally, to cross-compile just run:

docker run --network host -it --rm \
       -v "$(pwd)":"/home/docker/kvm-rust" \
       arm-rust-build \
       cargo build --release --target=aarch64-unknown-linux-gnu

First roadblock: API difference

At first, I thought the difference would be mostly in CPU registers, so I
removed KVM_GET_SREGS2 and KVM_SET_SREGS2 and some x86-specific code just to
make it compile. However, when I ran it on Raspberry Pi, the program returned an
error on KVM_GET_REGS. I once again turned to use strace on QEMU to see what
was to be done. I downloaded the Linux kernel source code (from www.kernel.org),
cross-compiled it for ARM and ran it in QEMU:

#!/bin/sh

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt                                                 \
        -smp 1                                                  \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0"                               \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults

Notice that here I used the virt machine specifically: that's the option to
use for KVM-based virtualization (machines like raspi4b would be emulated as
they require a very specific CPU and device tree, so they cannot simply run
directly on the host CPU). I put this into start-qemu.sh file and then ran:

strace -yy -f -X verbose -e trace=ioctl,openat,read,write,mmap -o kvm.log ./start-qemu.sh

Discovering KVM register API for ARM64

First thing I did was to try and find KVM_GET_REGS string, but there were no
results. I tried something more generic like grep -i regs and got this as
first results:

137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0
137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0
137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0
137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0
137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0
137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4010aeab /* KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG */, 0x7fff57e008) = 0

So this is when I realized it was going to be a bit harder. I wasn't getting
names of fields and corresponding values, just addresses. I installed Linux
headers on Raspberry Pi and searched for KVM_[GS]ET_ONE_REG:

pi@raspberrypi:~ $ grep -Rn KVM_[GS]ET_ONE_REG /usr/include/linux/
/usr/include/linux/kvm.h:1618:#define KVM_GET_ONE_REG             _IOW(KVMIO,  0xab, struct kvm_one_reg)
/usr/include/linux/kvm.h:1619:#define KVM_SET_ONE_REG             _IOW(KVMIO,  0xac, struct kvm_one_reg)

Searching in the same file we can see that the struct kvm_one_reg is defined
as follows:

struct kvm_one_reg {
       __u64 id;
       __u64 addr;
};

Constructing CPU id

As it usually is with great discoveries, I learned how to construct the register
id in kvm_one_reg by accident. Before even knowing how KVM on ARM works with
registers, I naively tried searching for struct kvm_regs and found two
interesting matches:

pi@raspberrypi:~ $ grep -Rn kvm_regs /usr/include/
/usr/include/aarch64-linux-gnu/asm/kvm.h:50:struct kvm_regs {
/usr/include/aarch64-linux-gnu/asm/kvm.h:208:#define KVM_REG_ARM_CORE_REG(name)(offsetof(struct kvm_regs, name) / sizeof(__u32))

The struct kvm_regs here was just a list of registers and
KVM_REG_ARM_CORE_REG macro seemed to be taking a register name as input and
looking for offset of the register in the struct kvm_regs, but I didn't find
it anywhere in the Linux headers.

I knew, however, that QEMU was one place where this might be used and I searched
for KVM_REG_ARM_CORE_REG and found this in target/arm/kvm.c:

#define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
                 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))

What was left to do was just to find these macro definitions and reconstruct
them in my Rust KVM code:

const KVM_REG_ARM64 : u64 = 0x6000000000000000;
const KVM_REG_SIZE_U64 : u64 = 0x0030000000000000;
const KVM_REG_ARM_COPROC_SHIFT : u64 = 16;
const KVM_REG_ARM_CORE : u64 = 0x0010 << KVM_REG_ARM_COPROC_SHIFT;

However, the KVM_REG_ARM_CORE_REG needed to be more thought-through as it
worked with offsets in the struct kvm_regs. So I have written a function
taking the register name as string and mapping it to the register id that KVM
understands:

fn AARCH64_CORE_REG(name: &str) -> io::Result<u64> {
    let base = KVM_REG_ARM64 | KVM_REG_SIZE_U64 | KVM_REG_ARM_CORE;
    if name.starts_with("x") {
        let reg : u64 = name[1..]
            .parse()
            .map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
        return Ok(base | reg * 2); }

    match name {
        "sp" => { return Ok(base | (31 * 2)); },
        "pc" => { return Ok(base | (32 * 2)); },
        "pstate" => { return Ok(base | (33 * 2)); },
        _ => ()
    };

    Err(io::Error::other("Invalid register."))
}

In my implementation I have improved this further by utilizing HashMap and
LazyLock so that the parsing and calculation isn't done on-the fly but rather
computed once and cached for subsequent calls.

Final thoughts

At this point I understood how ARM64 exposes CPU registers through
KVM_GET_ONE_REG and KVM_SET_ONE_REG calls and had reproduced the register
id encoding in Rust. Unfortunately, that wasn't enough to make the hypervisor
work. My first attempt at accessing the program counter (PC register)
immediately failed with an ENOEXEC error: a clue that I was missing another
crucial step in KVM ARM setup.

The code showcasing creation of register ids in a HashMap and subsequent
error when calling KVM_SET_ONE_REG can be found on my GitHub page:

https://github.com/StjepanPoljak/kvm-rust/tree/kvm-arm-part1-code

Building a KVM Virtual Machine in Rust: Running a binary

Sun, 28 Jun 2026 05:24:29 +0000

Recap

This is a continuation of Part 2 of KVM in Rust series,
where we successfully set up a memory region for our guest VM. Now we will see
how to proceed further and run a very simple binary in our hypervisor.

Loading the binary

Now, we have only set up a memory region, but we still haven't loaded any binary
or code to actually run. For this, I have written a small "Hello world" x86
assembly file:

https://github.com/StjepanPoljak/kvm-rust/blob/kvm-part3-code/samples/hello-world.asm

Notice that the org 0x1000 header is quite arbitrary and it's actually a good
lead of what we also need to do (set the instruction pointer). Let's compile the
assembly file with:

nasm ./samples/hello-world.asm

Then, we want to load our binary into the memory region at address 0x1000:

let mut file = File::open("./samples/hello-world")?;
let mut code = Vec::new();
file.read_to_end(&mut code)?;

unsafe {
std::ptr::copy_nonoverlapping(code.as_ptr(),
  (mem_ptr as *mut u8).add(0x1000),
  code.len()) };

Here we are using the more optimal copy_nonoverlapping because we can be quite
certain that the memory where code is stored is not overlapping with our
memory region.

Creating a vCPU

As we saw in Part 1, we have also discovered that QEMU also
calls KVM_SET_VCPU to obtain a vCPU file descriptor on which it will call
KVM_RUN. So, all that we have to do to set up a vCPU is to take a look at the
ioctl and recreate it in Rust:

// 140904 ioctl(9<anon_inode:kvm-vm>, 0xae41 /* KVM_CREATE_VCPU */, 0) = 10<anon_inode:kvm-vcpu:0>
let vcpu_fd = unsafe { libc::ioctl(self.fd, KVM_CREATE_VCPU, 0usize) };
if vcpu_fd < 0 {
return Err(io::Error::last_os_error());
}

Investigating vCPU operations

It would be tempting to simply call KVM_RUN on this file descriptor, however,
we don't really know how vCPU is set up, where the instruction pointer is and
even if the registers are in the valid state. If we look at the ioctl calls
referring to kvm-vcpu:0 in our log (and filtering out the spamming KVM_RUN
and KVM_SMI), we can see a lot of lines like these being repeated:

$ grep 'kvm-vcpu:0' kvm.log | grep -v 'RUN\|SMI'

                ### omitted a lot of repetitive strace output ###

140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xc080aebe /* KVM_GET_NESTED_STATE */, 0x7768f8002010) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xc028ae92 /* KVM_TPR_ACCESS_REPORTING */, 0x77690198f090) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4140aecd /* KVM_SET_SREGS2 */, 0x77690198f2f0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4090ae82 /* KVM_SET_REGS */, {rax=0x20, ..., rsp=0x6d88, rbp=0, ..., rip=0x18, rflags=0x246}) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x5000aea5 /* KVM_SET_XSAVE */, 0x7768f8001000) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4188aea7 /* KVM_SET_XCRS */, 0x77690198f2c0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4008ae89 /* KVM_SET_MSRS */, 0x7768f80040a0) = 59
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4040aea0 /* KVM_SET_VCPU_EVENTS */, 0x77690198f500) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4008ae89 /* KVM_SET_MSRS */, 0x7768f80040a0) = 1
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4080aea2 /* KVM_SET_DEBUGREGS */, 0x77690198f500) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xc008ae88 /* KVM_GET_MSRS */, 0x77690198f5a0) = 1

So we see that QEMU is setting vCPU registers via KVM_SET_REGS and in this log
we can see their exact values. It's also calling KVM_SET_SREGS2 which is
setting system registers; unfortunately, we cannot see the values used here.
Also, we do not need to follow the whole logic here, we just want to see what
QEMU is doing with registers before calling KVM_RUN:

$ awk '/KVM_RUN/ {print; exit} {print}' kvm.log | grep '[GS]ET_[S]\?REGS\|KVM_RUN'
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4140aecd /* KVM_SET_SREGS2 */, 0x77690198f310) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4090ae82 /* KVM_SET_REGS */, {rax=0, ..., rsp=0, rbp=0, ..., rip=0xfff0, rflags=0x2}) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x8090ae81 /* KVM_GET_REGS */, {rax=0, ..., rsp=0, rbp=0, ..., rip=0xfff0, rflags=0x2}) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x8140aecc /* KVM_GET_SREGS2 */, 0x77690198f310) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4140aecd /* KVM_SET_SREGS2 */, 0x77690198f310) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4090ae82 /* KVM_SET_REGS */, {rax=0, ..., rsp=0, rbp=0, ..., rip=0xfff0, rflags=0x2}) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = -1 EINTR (Interrupted system call)

Since QEMU first calls SET_SREGS2 and SET_REGS we should actually
investigate what the default (if any) values are in our own implementation.

Implementing register get/set operations in Rust

The procedure for this is as with all other KVM ioctl calls we implemented.
We find out how to construct their numbers (consult the previous article here)
and reimplement these constants in Rust:

const KVM_GET_SREGS2 : u64 = _IOR::<kvm_sregs2>(KVMIO, 0xcc);
const KVM_SET_SREGS2 : u64 = _IOW::<kvm_sregs2>(KVMIO, 0xcd);
const KVM_GET_REGS : u64 = _IOR::<kvm_regs>(KVMIO, 0x81);
const KVM_SET_REGS : u64 = _IOW::<kvm_regs>(KVMIO, 0x82);

We also need to add kvm_sregs2 and kvm_regs in our build.rs file:

    bindgen::Builder::default()
        .header("/usr/include/linux/kvm.h")
        .allowlist_type("kvm_sregs2")
        .allowlist_type("kvm_userspace_memory_region")
        .allowlist_type("kvm_regs")
        .generate_comments(false)
        .generate()?
        .write_to_file(out_path.join("kvm-bindings.rs"))?;

Then, ioctl calls are as follows (I will only give example for get_sregs2
and get_regs):

let mut sregs2 = unsafe { std::mem::zeroed() };
let ret = unsafe { libc::ioctl(self.fd, KVM_GET_SREGS2, &mut sregs2) };

let mut regs = unsafe { std::mem::zeroed() };
let ret = unsafe { libc::ioctl(self.fd, KVM_GET_REGS, &mut regs) };

Peeking at registers

After obtaining sregs2 and regs we can print them out and see what they are
set to by default (to accomplish this we consult the struct definition in our
kvm-bindings.rs (or in the Linux headers) and print out all the values of the
fields we find). For regs we see they are pre-set to:

RAX=0x0 RBX=0x0 RCX=0x0 RDX=0x600
RSI=0x0 RDI=0x0 RSP=0x0 RBP=0x0
R8=0x0  R9=0x0  R10=0x0 R11=0x0
R12=0x0 R13=0x0 R14=0x0 R15=0x0
RIP=0xfff0      RFLAGS=0x2

Taking a look at our trace from kvm.log we can see that the ones QEMU sets are
exactly the same (except for RDX but we can get away with this one).

Peeking at system registers

Now, for sregs2 it gets a bit more complicated. Our defaults are as follows:

CS      base=0xffff0000 selector=0xf000 limit=0xffff    type=0xb        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

DS      base=0x0        selector=0x0    limit=0xffff    type=0x3        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

ES      base=0x0        selector=0x0    limit=0xffff    type=0x3        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

FS      base=0x0        selector=0x0    limit=0xffff    type=0x3        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

GS      base=0x0        selector=0x0    limit=0xffff    type=0x3        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

SS      base=0x0        selector=0x0    limit=0xffff    type=0x3        present=0x1
        dpl=0x0         db=0x0          s=0x1   l=0x0   g=0x0           avl=0x0

TR      base=0x0        selector=0x0    limit=0xffff    type=0xb        present=0x1
        dpl=0x0         db=0x0          s=0x0   l=0x0   g=0x0           avl=0x0

LDT     base=0x0        selector=0x0    limit=0xffff    type=0x2        present=0x1
        dpl=0x0         db=0x0          s=0x0   l=0x0   g=0x0           avl=0x0

GDT     base=0x0        limit=0xffff

IDT     base=0x0        limit=0xffff

CR0=0x60000010          CR2=0x0         CR3=0x0         CR4=0x0         CR8=0x0

EFER=0x0                APIC_BASE=0xfee00900            FLAGS=0x0

PDPTRS[0]=0x0           PDPTRS[1]=0x0           PDPTRS[2]=0x0           PDPTRS[3]=0x0

Problem is that we don't see any values in our strace log, only the address of
the variable. So we need to be a bit creative here if we want to find out what
QEMU sets these values to. We could use ptrace (in fact strace is built upon
ptrace API), but it may be a bit too much. Same for uprobes and eBPF. We
do have GDB, though, and it's just perfect as a one-off thing here. All we have
to do is run QEMU under GDB and then execute:

(gdb) break ioctl if $rsi == 0x4140aecd
(gdb) run

Note that 0x4140aecd is the exact value that we extracted from SET_SREGS2
ioctl call (in x86 ABI, RSI is holding the value of second argument):

140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4140aecd /* KVM_SET_SREGS2 */, 0x77690198f310) = 0

Once GDB breaks, we can dump memory. However, we don't know the exact size of
struct kvm_sregs2; we could guess or manually inspect, but quickest way to get
it is to actually have a small C program:

#include <linux/kvm.h>
#include <stdlib.h>
#include <stdio.h>

int main() {
  printf("%lu", sizeof(struct kvm_sregs2));
  return 0;
}

The program will return value of 320 (at least on my system) and we can then
use this in GDB:

(gdb) dump memory dump.bin $rdx $rdx+320

Note that RDX is holding the third argument, i.e. the address we actually saw
in our strace log. Now, we can reuse our function for printing sregs2 in
Rust. We simply hack our main function to load the file and reinterpret its data
as sregs2 and then print it. We got:

CSbase=0xffff0000selector=0xf000limit=0xfffftype=0xbpresent=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

DSbase=0x0selector=0x0limit=0xfffftype=0x3present=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

ESbase=0x0selector=0x0limit=0xfffftype=0x3present=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

FSbase=0x0selector=0x0limit=0xfffftype=0x3present=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

GSbase=0x0selector=0x0limit=0xfffftype=0x3present=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

SSbase=0x0selector=0x0limit=0xfffftype=0x3present=0x1
dpl=0x0db=0x0s=0x1l=0x0g=0x0avl=0x0

TRbase=0x0selector=0x0limit=0xfffftype=0xbpresent=0x1
dpl=0x0db=0x0s=0x0l=0x0g=0x0avl=0x0

LDTbase=0x0selector=0x0limit=0xfffftype=0x2present=0x1
dpl=0x0db=0x0s=0x0l=0x0g=0x0avl=0x0

GDTbase=0x0limit=0xffff

IDTbase=0x0limit=0xffff

CR0=0x60000010CR2=0x0CR3=0x0CR4=0x0CR8=0x0

EFER=0x0APIC_BASE=0xfee00900FLAGS=0x0

PDPTRS[0]=0x0PDPTRS[1]=0x4b275f5fce32f200PDPTRS[2]=0x0PDPTRS[3]=0x3

We can see that this more or less corresponds to the defaults we observed in our
KVM implementation. If we want to change our registers (which we will), then we
will always first get the ones that are current in the vCPU, change the ones we
need and then set them using KVM_SET_REGS (or KVM_SET_SREGS2). So one of
first things to do is to set CS to flat mapping:

let mut sregs2 = vcpu.get_sregs2()?;
sregs2.cs.base = 0;
sregs2.cs.selector = 0;
vcpu.set_sregs2(sregs2)?;

Recall that our assembly file was assembled with org 0x1000, so setting RIP to
0x1000 causes execution to begin at the start of the loaded binary:

let mut regs = vcpu.get_regs()?;
regs.rip = 0x1000;
vcpu.set_regs(regs)?;

Furthermore, we now have a very good process of finding out what QEMU does with
registers.

Running the binary

Now this is where strace stopped being useful and my understanding of the KVM
run loop became more useful. Another positive thing is that this part of QEMU
source code is quite readable and can be found in function kvm_vcpu_thread_fn.

The more x86-specific code with exit reasons can be found in the kvm_arch_handle_exit function.

Setting kvm_run shared memory region

To wrap things up, first we need to mmap the shared kvm_run structure that KVM
uses to communicate VM-exit information and other runtime state between the
kernel and userspace:

// 140904 ioctl(3</dev/kvm<char 10:232>>, 0xae04 /* KVM_GET_VCPU_MMAP_SIZE */, 0) = 12288
let kvm_run_size = unsafe {
libc::ioctl(kvm_fd, KVM_GET_VCPU_MMAP_SIZE, 0usize) };
let kvm_run_mem = unsafe {
libc::mmap(ptr::null_mut(), kvm_run_size as usize,
   libc::PROT_READ|libc::PROT_WRITE, libc::MAP_SHARED,
   vcpu_fd, 0) };

Then, when we call KVM_RUN we simply match the exit reasons and act
accordingly:

let run = vcpu.kvm_run_mem as *mut kvm_run;

loop {
    let ret = unsafe { libc::ioctl(vcpu.fd, KVM_RUN, 0usize) };
    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    let exit_reason = unsafe { (*run).exit_reason };

    match exit_reason {
        KVM_EXIT_MMIO => { /* omitted */ }
        KVM_EXIT_HLT => {
            println!("Guest halted.");
            break; }
        KVM_EXIT_SHUTDOWN => {
            println!("Guest shutdown.");
            break; }
        KVM_EXIT_INTERNAL_ERROR => {
            return Err(io::Error::other("KVM internal error.")); }
        _ => {
            println!("EXIT REASON = {}", exit_reason);
        }
    }
}

Our first output

Being that our assembly file uses VGA MMIO to output the "Hello world!" string,
I have only implemented KVM_EXIT_MMIO. Then we can finally see the output:

$ cargo run ./samples/hello-world
     ## ommitted sregs2 output and warnings ##
RAX=0x0 RBX=0x0 RCX=0x0 RDX=0x600
RSI=0x0 RDI=0x0 RSP=0x0 RBP=0x0
R8=0x0  R9=0x0  R10=0x0 R11=0x0
R12=0x0 R13=0x0 R14=0x0 R15=0x0
RIP=0x1000      RFLAGS=0x2
Hello from KVM!
Guest halted.
RAX=0xb800      RBX=0x0 RCX=0x0 RDX=0x600
RSI=0x102d      RDI=0x22        RSP=0x0 RBP=0x0
R8=0x0  R9=0x0  R10=0x0 R11=0x0
R12=0x0 R13=0x0 R14=0x0 R15=0x0
RIP=0x101b      RFLAGS=0x46

Conclusion

This concludes the three-part series on figuring out KVM via strace and
reimplementing it in Rust. A lot of next steps actually come down to x86
architecture and bootloader specifics, so we may leave it here for now. The full
code capable of running simple binaries can be found on GitHub:

https://github.com/StjepanPoljak/kvm-rust/tree/kvm-part3-code

Building a KVM Virtual Machine in Rust: Memory Setup

Mon, 15 Jun 2026 05:20:13 +0000

Recap

This is a continuation of my previous article which dealt
with reverse-engineering QEMU with strace to learn how KVM works. Now it's
time to try and follow the steps we got from the strace logs to build our own
KVM-based virtual machine in Rust.

KVM Headers

I haven't actually used existing KVM libraries written specifically for Rust but
opted to use the libc crate which provides the required ioctl bindings and
helper macros. The main reason is that I want a complete understanding of what
is happening under the hood. Now for each of these KVM ioctl calls we can use
the Linux headers for reference. For example, to find out how to construct
ioctl number for KVM_CREATE_VM we simply can do:

$ grep -Rn 'KVM_CREATE_VM' /usr/include/linux/
/usr/include/linux/kvm.h:855:/* machine type bits, to be used as argument to KVM_CREATE_VM */
/usr/include/linux/kvm.h:882:#define KVM_CREATE_VM             _IO(KVMIO,   0x01) /* returns a VM fd */

Luckily in libc Rust crate we have macros for _IO (and the like), but we
still need KVMIO macro:

$ grep -Rn 'define KVMIO' /usr/include/linux/
/usr/include/linux/kvm.h:853:#define KVMIO 0xAE

We can now construct the ioctl number for KVM_CREATE_VM:

use libc::{_IOW, _IO, _IOR};

const KVMIO : u32 = 0xae;
const KVM_CREATE_VM : u64 = _IO(KVMIO, 0x01);

Note that exact integer type depends on the platform and libc definitions.
Then, in the main we can do:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let file = OpenOptions::new()
        .read(true)
        .write(true)
        .open("/dev/kvm")
        .expect("failed to open /dev/kvm");

    let fd = file.as_raw_fd();
    let vm_fd = unsafe { libc::ioctl(fd, KVM_CREATE_VM, 0usize) };
Ok(())
}

This is the procedure we will follow for each relevant ioctl call. In fact,
we can "reverse-engineer" our own program and then compare it with the original,
to make sure we are doing the right thing:

$ strace cargo run

     ### omitted irrelevant strace output ###

openat(AT_FDCWD, "/dev/kvm", O_RDWR|O_CLOEXEC) = 3
ioctl(3, KVM_CREATE_VM, 0)              = 4

Important note

In production code we would immediately check for a negative return value and
convert errno into a Rust error. To keep the example focused, I am omitting
proper error handling in this article.

Setting memory region

Now, to recall, the next step is setting up the memory region used by both the
KVM guest and the host. This region will be the memory of our virtual machine, a
place where we will load our binary:

140900 mmap(NULL, 1075838976, 0 /* PROT_NONE */, 0x22 /* MAP_PRIVATE|MAP_ANONYMOUS */, -1, 0) = 0x7768b3e00000
140900 mmap(0x7768b3e00000, 1073741824, 0x3 /* PROT_READ|PROT_WRITE */, 0x32 /* MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS */, -1, 0) = 0x7768b3e00000
140900 ioctl(9<anon_inode:kvm-vm>, 0x4020ae46 /* KVM_SET_USER_MEMORY_REGION */, {slot=0, flags=0, guest_phys_addr=0, memory_size=1073741824, userspace_addr=0x7768b3e00000}) = 0

Recreating mmap call

So, first thing we need to do is follow the same logic for mmap which is also
available in the libc Rust crate. After creating the virtual machine, we could
simply recreate our own mmap calls based on the strace output. However,
notice that QEMU first reserves a larger address range with PROT_NONE and then
maps only the portion it actually intends to use. For our prototype we do not
actually need to mimic this exact reservation pattern.

let mem_size: u64 = 256 * 1024;
let mem = unsafe {
libc::mmap(ptr::null_mut(),
   mem_size as usize,
   libc::PROT_READ|libc::PROT_WRITE,
   libc::MAP_PRIVATE|libc::MAP_ANONYMOUS,
   -1,
   0)
};

For this experiment we are only allocating 256 kilobytes because we are not yet
booting a full operating system and therefore need very little guest memory.
Also note that, as with ioctl, production code should check whether mmap()
returned MAP_FAILED.

Recreating ioctl call

First we need to see the definition of the KVM_SET_USER_MEMORY_REGION:

$ grep -Rn 'define KVM_SET_USER' /usr/include/linux/ -A1
/usr/include/linux/kvm.h:1433:#define KVM_SET_USER_MEMORY_REGION _IOW(KVMIO, 0x46, \
/usr/include/linux/kvm.h-1434-                                  struct kvm_userspace_memory_region)

We see that for this one, we need struct kvm_userspace_memory_region. The
values we need are visible in the strace output, but copying struct definition
to our Rust program is really not advisable. Luckily, Rust has bindgen which
we can use to get this KVM struct (and others) from Linux headers. For this
purpose we have a separate build.rs file which will contain:

use bindgen;
use std::path::PathBuf;
use std::env;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let out_path = PathBuf::from(env::var("OUT_DIR")?);

    bindgen::Builder::default()
.header("/usr/include/linux/kvm.h")
.allowlist_type("kvm_userspace_memory_region")
.generate_comments(false)
.generate()?
.write_to_file(out_path.join("kvm-bindings.rs"))?;

    Ok(())
}

Now in the main.rs we can import these bindings with:

include!(concat!(env!("OUT_DIR"), "/kvm-bindings.rs"));

Then, after mmap() calls, we set the memory region, also imitating what
QEMU is doing in the strace output:

let region = kvm_userspace_memory_region {
slot : 0,
flags : 0,
guest_phys_addr : 0x0,
memory_size : mem_size,
userspace_addr : mem as u64
};

let _ret = unsafe { libc::ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region) };

We register this region starting at guest physical address 0x0, meaning the
first byte of our allocated host memory will appear as physical address 0 inside
the guest. Also note that the KVM_SET_USER_MEMORY_REGION call does not copy
memory, but rather tells KVM that guest physical address will be backed by a
specific userspace memory region.

Running the code

Next thing to do is to run it. We still haven't added any checks after mmap
and ioctl calls, but for this prototype we can again simply use strace our
own code:

$ strace -yy -X verbose -e trace=ioctl,mmap,openat,read,write cargo run

                ### omitted irrelevant strace output ###

openat(-100 /* AT_FDCWD */</home/stjepan/Develop/KVM/rust>, "/dev/kvm", 0x80002 /* O_RDWR|O_CLOEXEC */) = 3</dev/kvm<char 10:232>>
ioctl(3</dev/kvm<char 10:232>>, 0xae01 /* KVM_CREATE_VM */, 0) = 4<anon_inode:kvm-vm>
mmap(NULL, 262144, 0x3 /* PROT_READ|PROT_WRITE */, 0x22 /* MAP_PRIVATE|MAP_ANONYMOUS */, -1, 0) = 0x7d63fdfa2000
ioctl(4<anon_inode:kvm-vm>, 0x4020ae46 /* KVM_SET_USER_MEMORY_REGION */, {slot=0, flags=0, guest_phys_addr=0, memory_size=262144, userspace_addr=0x7d63fdfa2000}) = 0

We can see our output is fine and no errors were reported. Note that a full
working example with proper checking and Rust idiomatic approaches can be found
on my GitHub page:

https://github.com/StjepanPoljak/kvm-rust/tree/kvm-part2-code

Next steps

At this point we have a VM object and guest memory, but nothing is actually executing yet. In the next part we will create a vCPU, initialize its state, load a small binary into guest memory and enter the first KVM_RUN loop.

Learning KVM by Reverse-Engineering QEMU with strace

Thu, 11 Jun 2026 15:56:48 +0000

Motivation

I work with virtual machines in QEMU/KVM environment (a lot). In order to debug,
optimize and customize the VMs requires an in-depth knowledge of both QEMU and
KVM, the Linux kernel virtualization subsystem that exposes hardware
virtualization features such as Intel VT-x and AMD-V to userspace applications
like QEMU. Not only that, but I work on a lot of hobby projects requiring quick
bare-metal boot-ups and debugging workflows, and, to be honest, a lot of times
QEMU is an overkill for these sorts of tasks.

KVM vs TCG

Also when it comes to QEMU, it's worth noting that we always have an option of
using TCG, which has a completely different purpose than KVM. TCG is short for
Tiny Code Generator, which works by translating guest instructions into host
instructions at runtime. This is quite slow compared to running code without
translation overhead. So, if we want to test our bare-metal code, we may want to
test it on our own real CPU at native speed. This is where KVM comes in. Unlike
TCG, KVM does not emulate the CPU itself. Instead, it allows guest code to
execute directly on the host processor while the Linux kernel manages
transitions between guest and host execution.

How KVM works in Linux

So, I do already know some basics. KVM driver exposes a driver interface in
Linux root filesystem, /dev/kvm. Communicating with the driver is done via
ioctl() system call on a file descriptor. What we need to find out is how QEMU
communicates with Linux kernel and try and follow the QEMU logic without
reading QEMU source code and KVM API (both can be a bit more intimidating than
just seeing how it works under the hood).

Reverse-engineering KVM

Now we can take some lightweight Debian Linux image and load it into the QEMU,
with KVM enabled:

QEMU_IMAGE=./debian-12-nocloud-amd64.qcow2

qemu-system-x86_64\
-m 1024\
-drive file="${QEMU_IMAGE}",if=virtio,cache=none\
-serial stdio\
-enable-kvm\
-cpu host\
-nodefaults\
-nographic

This is a pretty straightforward way to run QEMU with minimal setup. The most
relevant options for us are -enable-kvm and -cpu host, which will enable
KVM and use host CPU instead of emulating some specific CPU.

Tracing QEMU/KVM with strace

Now, we want to see what QEMU is really doing by utilizing strace. We can put this command in a start-qemu.sh script and call it with:

strace -yy -f -X verbose\
   -e trace=ioctl,openat,read,write,mmap\
   -o kvm.log\
   ./start-qemu.sh

This command will trace all ioctl, openat, read, write and mmap system
calls. Although I mentioned only ioctl calls so far, I always like to include
some other common system calls that could be used. As far as we know /dev/kvm
is the interface to KVM driver and QEMU will probably use openat on it.
Similarly, we also want to see what QEMU is doing with memory and what it's
reading and writing in general.

Note: Information on strace arguments as above can be found in strace --help
or man strace, but essentially, the -yy tells strace to print all available
information when decoding file descriptors, -f follows forks (we need this one
as we're wrapping it in scripts and QEMU might also do similar stuff). The
-X verbose will print names of constants and flags (very important when
analyzing ioctl calls);

Interpreting the logs

Now, we start the above command and, as soon as system boots, we can kill it
with CTRL+C. This will be quite sufficient to see how QEMU/KVM works without
spamming our logs with redundant information. When we read the kvm.log file,
we will see a lot of traces that are not really interesting. However, we already
have some knowledge: we know QEMU should be opening /dev/kvm so a quick
search for kvm reveals exactly what we need:

140900 openat(-100 /* AT_FDCWD */</home/stjepan/Develop/KVM>, "/dev/kvm", 0x80002 /* O_RDWR|O_CLOEXEC */) = 3</dev/kvm<char 10:232>>
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae00 /* KVM_GET_API_VERSION */, 0) = 12
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae03 /* KVM_CHECK_EXTENSION */, 0x88 /* KVM_CAP_IMMEDIATE_EXIT */) = 1
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae03 /* KVM_CHECK_EXTENSION */, 0xa /* KVM_CAP_NR_MEMSLOTS */) = 32764
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae03 /* KVM_CHECK_EXTENSION */, 0x76 /* KVM_CAP_MULTI_ADDRESS_SPACE */) = 2
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae01 /* KVM_CREATE_VM */, 0) = 9<anon_inode:kvm-vm>
140900 ioctl(9<anon_inode:kvm-vm>, 0xae03 /* KVM_CHECK_EXTENSION */, 0x9 /* KVM_CAP_NR_VCPUS */) = 4
140900 ioctl(3</dev/kvm<char 10:232>>, 0xae03 /* KVM_CHECK_EXTENSION */, 0x42 /* KVM_CAP_MAX_VCPUS */) = 4096

We can see that QEMU is opening /dev/kvm and that it's checking API version
and various extensions. We may skip these checks and focus on the calls that
look most important; one of these here is KVM_CREATE_VM which also returns a
file descriptor 9<anon_inode:kvm-vm> which we can use as a further reference.

Setting up memory regions

We know QEMU must eventually load firmware and guest memory into the VM. Looking
for file operations after KVM_CREATE_VM, we quickly encounter SeaBIOS being
loaded:

140900 openat(-100 /* AT_FDCWD */</home/stjepan/Develop/KVM>, "/usr/share/seabios/bios-256k.bin", 0 /* O_RDONLY */) = 12</usr/share/seabios/bios-256k.bin>
140900 mmap(NULL, 2359296, 0 /* PROT_NONE */, 0x22 /* MAP_PRIVATE|MAP_ANONYMOUS */, -1, 0) = 0x776900dc1000
140900 mmap(0x776900e00000, 262144, 0x3 /* PROT_READ|PROT_WRITE */, 0x32 /* MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS */, -1, 0) = 0x776900e00000
140900 openat(-100 /* AT_FDCWD */</home/stjepan/Develop/KVM>, "/usr/share/seabios/bios-256k.bin", 0 /* O_RDONLY */) = 12</usr/share/seabios/bios-256k.bin>
140900 mmap(NULL, 266240, 0x3 /* PROT_READ|PROT_WRITE */, 0x22 /* MAP_PRIVATE|MAP_ANONYMOUS */, -1, 0) = 0x776900fc0000
140900 read(12</usr/share/seabios/bios-256k.bin>, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144) = 262144

We can see QEMU opening and reading SeaBIOS binary and reserving memory along
with it. We can use the return of the read system call (the size of the
SeaBIOS binary) and see what KVM is doing with it. Searching through the log
for this size gives us further information on what KVM is doing with SeaBIOS:

140900 ioctl(9<anon_inode:kvm-vm>, 0x4020ae46 /* KVM_SET_USER_MEMORY_REGION */, {slot=3, flags=0x2 /* KVM_MEM_READONLY */, guest_phys_addr=0xfffc0000, memory_size=262144, userspace_addr=0x776900e00000}) = 0

So we see that it's now setting this as the memory region for KVM at guest
physical address 0xfffc0000 from userspace address that was actually
obtained by mmap in one of the traces above. In other words, KVM does not
allocate guest RAM itself; userspace applications such as QEMU remain
responsible for managing the backing memory.

Creating vCPU and running

Now, it gets very busy in the logs, but most of the stuff we see is still just
checking for extensions and capabilities. However, if we take a look at the
tail of the log, we will see a lot of these ioctl calls:

140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xaeb7 /* KVM_SMI */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xae80 /* KVM_RUN */, 0) = 0
140904 ioctl(10<anon_inode:kvm-vcpu:0>, 0xaeb7 /* KVM_SMI */, 0) = 0

Now both KVM_RUN and KVM_SMI are operating on a kvm-vcpu file descriptor,
something we haven't yet seen. So if we search the logs for it, we can actually
see where it's created:

140904 ioctl(9<anon_inode:kvm-vm>, 0xae41 /* KVM_CREATE_VCPU */, 0) = 10<anon_inode:kvm-vcpu:0>

Conclusion

Now we have a more complete picture of how QEMU is setting up KVM. First,
/dev/kvm is opened to obtain a file descriptor representing the KVM subsystem.
From it we create a new virtual machine and get kvm-vm file descriptor. On
this file descriptor we are setting up memory regions and later use it to create
a vCPU, on which we can call KVM_RUN. The following diagram explains it
better:

/dev/kvm   kvm fd
    |
    +--> KVM_CREATE_VM   kvm-vm fd
            |
            +--> KVM_SET_USER_MEMORY_REGION
            |
            +--> KVM_CREATE_VCPU   kvm-vcpu fd
                            |
                            +--> KVM_RUN (loop)

As we can see from the logs, KVM_RUN appears repeatedly during guest
execution, while occasional KVM_SMI calls inject System Management Interrupts
into the guest. This repeated interaction between userspace and KVM is what
ultimately drives virtual CPU execution.

Next time we will recreate this exact behavior in Rust and also see about just
a few missing pieces to get our first virtual machine running in KVM.