Coder Legion - Recent posts tagged arm

Booting ARM64 Linux in my Rust KVM Hypervisor: Debugging a Memory Issue

Sun, 06 Sep 2026 18:39:35 +0000

Motivation

The time has come: the big test. I got some simple binaries running on my Rust
KVM hypervisor (at this point I am thinking about giving it a proper name). The
next thing has to be Linux. For this series I will show how to actually boot
Linux on my Rust KVM hypervisor on ARM64 CPU (I am using Raspberry Pi 4 which
supports hardware accelerated virtualization).

Cross-compiling Linux kernel

In one of my previous articles I have described how to
set up the cross-compilation environment with a simple Dockerfile. We can
reuse it to cross-compile the Linux kernel for ARM64 (after adding some Linux
build dependencies like bc, flex, python3, libssl-dev and bison):

docker run --network host -it --rm \
       -e CROSS_COMPILE=aarch64-linux-gnu- -e ARCH=arm64 \
       -v "$(pwd)":"/home/docker/kvm-rust" -w "/home/docker/kvm-rust/linux-7.1.3" \
       arm-rust-build /bin/bash -c 'make defconfig && make'

The resulting ARM64 Linux kernel image (Image) should be located in
arch/arm64/boot/ directory.

ARM64 boot protocol

The full ARM64 Linux boot protocol is quite straightforward and can be found in
the Linux kernel documentation:

https://www.kernel.org/doc/html/v7.1/arch/arm64/booting.html

We simply need to parse the 64-byte header and verify the magic field (it should
equal 0x644d5241). I will just highlight a few important points. We should
read image_size and text_offset fields and note that the image should be
loaded at an offset of text_offset bytes at a 2 MiB boundary. For our kernel,
as I checked, text_offset is zero and image_size is the amount of memory the
decompressed kernel image occupies in RAM, which is not necessarily the same as
the file size as reported by stat. Finally, the registers x1, x2 and x3
must be set to zero, while x0 must point to the device tree blob (dtb) in
the RAM. So, for this I chose to load the dtb at address 0x40000000 and
Image itself on the first 2 MiB boundary:

const LOAD_ADDR: u64 = 0x40_000_000;
const KERN_OFFS: u64 = 0x200_000;

Obtaining the device tree blob

So, for ARM64 we do need a device tree blob that will tell the Linux kernel on
what kind of hardware it is running on. And this step is quite easy, we will
can simply reuse our command for running QEMU, with one small addition (i.e.
adding dumpdtb option to -M virt argument):

#!/bin/sh

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt,dumpdtb=virt.dtb                                \
        -smp 1                                                  \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0"                               \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults

This gives us a consistent snapshot of the hardware we were running on (in this
case it's emulated hardware for QEMU virt machine). We could (and we should)
edit and prune it for our own purposes, but let's leave that for another time.

Implementing boot in Rust

So, loading the kernel in Rust is quite straightforward:

impl VM {
    pub fn arch_load_linux(&mut self, vcpu: &mut VCPU, args: &Args) -> io::Result<()> {
        let header = get_linux_header(&args.binary)?;
        let linux_mem_idx = self.add_mem_region((KERN_OFFS + header.image_size) as usize, LOAD_ADDR)?;

        match &args.dtb {
            Some(dtb_path) => { self.load_file_to_memory(linux_mem_idx, &dtb_path, 0x0)?; },
            None => { return Err(io::Error::other("Device tree blob not specified.")); }
        }

        self.load_file_to_memory(linux_mem_idx, &args.binary, KERN_OFFS)?;

        vcpu.set_one_reg("x0", LOAD_ADDR)?;
        vcpu.set_one_reg("x1", 0x0)?;
        vcpu.set_one_reg("x2", 0x0)?;
        vcpu.set_one_reg("x3", 0x0)?;

        vcpu.set_one_reg("pc", LOAD_ADDR + KERN_OFFS)?;
    }
}

Note that the get_linux_header() function was omitted here as it only parses
the aforementioned ARM64 Linux image headers in a boilerplate manner.

Function not implemented

As soon as I tried to run my hypervisor with Linux Image, I ran into an issue:

pi@raspberrypi:~ $ ./rust -d virt Image 
x0 = 0x0
pc = 0x1000
magic 0x644d5241, image_size: 38731776, file_size: 38040064, text_offset: 0x0
Loading "virt" to 0x0...
Loading "Image" to 0x200000...
x0 = 0x40000000
pc = 0x40200000
x0 = 0xfffffbfffda35000
pc = 0xffffd902f0f8a42c
Error: Os { code: 38, kind: Unsupported, message: "Function not implemented" }

Trying strace

First thing I turned to was strace; it's easy to use and gets some nice
information really fast:

ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee98768) = 0
x0 = 0x40000000
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee98768) = 0
pc = 0x40200000
ioctl(5, KVM_RUN, 0)                    = -1 ENOSYS (Function not implemented)
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee988a8) = 0
x0 = 0xfffffbfffda35000
ioctl(5, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fcee988a8) = 0
pc = 0xffffd902f0f8a42c
Error: Os { code: 38, kind: Unsupported, message: "Function not implemented" }
+++ exited with 1 +++

I didn't learn much other than the fact that KVM_RUN failed completely with
ENOSYS error code. My first attempt to find out what was going on was
searching the ARM64 KVM source code for ENOSYS in Linux repository. It didn't
take long, as it appeared only in one place which was a promising lead:

https://elixir.bootlin.com/linux/v7.1.3/source/arch/arm64/kvm/mmio.c#L188

As we can see, the function in question is io_mem_abort() in the Linux kernel.
Now, strace got us that far, but this is a Linux kernel function and not much
we can find out more about it with strace (which traces only system calls like
ioctl). I haven't even considered GDB because this was the host kernel,
and this would require a really complex time-consuming setup. Luckily, there was
a much easier way.

Analyzing IO memory abort with eBPF

The first thing I did was to list kprobes in bpftrace and see if I can hook
into the io_mem_abort at all via eBPF:

pi@raspberrypi:~ $ sudo bpftrace -l 'kprobe:io_mem_abort'
kprobe:io_mem_abort

I was in luck, so I could at least confirm that this function was the one
causing the issue, so I attached a kretprobe to it, ran bpftrace alongside
my hypervisor and got the confirmation I needed:

pi@raspberrypi:~ $ sudo bpftrace -e '
>    kretprobe:io_mem_abort {
>        printf("ret=%d\n", retval);
>    }'
Attaching 1 probe...
ret=-38

Finding out what really happened

Now that I have confirmed io_mem_abort was the culprit, it was time to
analyze it more deeply. I attached an additional kprobe so I could print both
the return value and the fault_ipa argument:

int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)

The updated bpftrace script printed the fault address.

pi@raspberrypi:~ $ sudo bpftrace -e '
>    kretprobe:io_mem_abort {
>        printf("ret=%d\n", retval);
>    }
>    kprobe:io_mem_abort {
>        printf("0x%llx\n", arg1);
>    }'
Attaching 2 probes...
0x47fff000
ret=-38

So the result surprised me as I initially suspected it was trying to access some
device that I haven't created; however, I realized that this was actually just
beyond the memory I had allocated for the Linux kernel to use. So it clicked: I probably hadn't allocated enough guest memory. Instead of allocating just
header.image_size + KERN_OFFS I increased it to 1 GiB:

let linux_mem_idx = self.add_mem_region(1024 * 1024 * 1024, LOAD_ADDR)?;

Conclusion

With that adjustment, the ENOSYS error disappeared. The guest was clearly
executing code, although I still wasn't seeing any output on the serial console.
That meant the memory issue was solved, but there was at least one more thing
left before we would actually see Linux boot.

AArch64 Register Conventions: What Survives a Function Call—and Why

Thu, 03 Sep 2026 20:13:24 +0000

What happens to a register when you call a function?

On AArch64, the answer isn't "the CPU saves it." When you call a function, the CPU doesn't look at a register table and decide what to preserve. BL doesn't know about caller-saved or callee-saved registers. Those are ABI rules imposed on software, not behavior baked into the hardware.

Once you understand that distinction, the rest of the calling convention becomes much easier to reason about: the ABI tells the caller and callee who is responsible for preserving what. That's why some registers are caller-saved, some are callee-saved, and why a register's ABI category says nothing about how long a particular value will live.

When you write C++ code, the compiler decides whether a value lives in a register, on the stack, or nowhere at all after optimization. But under the hood, most scalar arithmetic and logical operations operate on register operands - and understanding the calling convention can make the difference between code that runs and code that performs.

If you've ever looked at compiler output for AArch64 and wondered why the compiler chose specific registers for specific values, this post is for you.

AArch64 vs ARM64: what's in a name?

AArch64 is Arm's official name for the 64-bit execution state and instruction set architecture introduced with Armv8-A. ARM64 is the colloquial term used in operating systems, toolchains, and developer communities (Linux refers to it as aarch64 in kernel sources, Apple calls it arm64). They refer to the same thing. This post uses AArch64 to match the official architecture documentation, but you'll encounter both names in the wild.

The big picture

AArch64 has 31 general-purpose registers (X0–X30), plus the special-purpose stack pointer SP and 32 128-bit SIMD/floating-point registers. That's a lot of registers - and beginners often see "32" in architecture documentation, so it's worth clarifying up front: there are 31 general-purpose registers, plus SP, which is a special-purpose register rather than an ordinary GPR (more on that later).

But not all registers are created equal. The calling convention - the set of rules that govern how functions pass arguments and return values - assigns specific roles to specific registers. Understanding this convention requires a key distinction that underpins everything else.

Caller-saved vs callee-saved: the conceptual key

The ABI divides registers into two categories based on what happens across a function call:

Caller-saved (also called caller-clobbered): The caller cannot assume these registers survive a call. If the caller needs a value in one of these registers after a call, it must save the value before the call and restore it afterward. The called function is free to overwrite them.
Callee-saved: If the callee modifies one of these registers, it must save the original value and restore it before returning. The caller can safely assume these registers are preserved across the call.

Keep these names in mind as obligations, not characteristics:

"Caller-saved" means the caller must protect a value if it needs it after a call.
"Callee-saved" means the callee must restore the incoming value before returning if it modifies the register.

Caller-saved and callee-saved are ABI properties, not hardware properties. The processor does not enforce these conventions; they are rules that separately compiled functions agree to follow. The CPU is perfectly happy to let any function overwrite X19 - it's the calling convention that says "if you modify it, you must restore it before returning."

A common misconception

A callee-saved register isn't inherently a "long-lived register." It is simply a register whose incoming value must survive a call if the callee modifies it. The compiler can put a short-lived value in X19, and it can put a long-lived value in X9 if that value doesn't cross a call. Register allocation is driven by liveness, register pressure, call boundaries, and cost models - not by which category a register falls into.

To make this precise, keep three separate concepts in mind:

Register volatility: what happens to the register across a call (caller-clobbered vs callee-saved).
Value liveness: whether a particular value is needed again later in the code.
Register allocation: which physical register the compiler chooses for a value.

These are three different concepts. A register can be caller-clobbered while holding a long-lived value, provided no call occurs while that value is live. A callee-saved register can hold a short-lived value. "Callee-saved" describes the register's contract, not the lifetime of the value stored in it.

Scope: what this article covers

This article focuses on the core AArch64 GPR and SIMD/FP register conventions used in ordinary AAPCS64 code. The current AAPCS64 also defines calling conventions for SVE and SME state - the Z0–Z31 scalable vector registers, P0–P15 predicate registers, the FFR, and the SME ZA/ZT0/FPMR state. These are outside the scope of this introduction.

The calling-convention register map

Here's the calling-convention register map as a quick overview. We'll walk through each group in detail below - remember, the full reference table lives in the cheat sheet at the end:

X0–X7 — Caller-clobbered: integer/pointer arguments and results
X8 — Caller-clobbered: indirect result location
X9–X15 — Caller-clobbered: temporary/scratch
X16–X17 — Caller-clobbered: IP0/IP1, linker/veneer scratch
X18 — Platform-dependent: platform register or caller-clobbered
X19–X28 — Callee-saved: general-purpose preserved state
X29 — Callee-saved: frame pointer / general-purpose register
X30 — Special: link register
SP — Callee-saved / special: stack pointer
V0–V7 — Caller-clobbered: FP/SIMD args and results
V8–V15 — Callee-saved, low 64 bits only: preserved FP/SIMD state
V16–V31 — Caller-clobbered: temporary FP/SIMD state
NZCV — Undefined: condition flags

X0–X7: arguments and results

X0–X7 are the primary general-purpose argument and result registers. For integer, pointer, and other appropriately classified arguments, the first eight argument-register slots use X0–X7 - note that "eight registers" does not necessarily mean "eight source-level arguments," because the AAPCS64 argument classification rules can cause arguments to be packed or passed differently. Floating-point and vector arguments normally go in V0–V7 instead, and aggregates can have more complex treatment depending on the ABI and argument types.

These registers are caller-saved (caller-clobbered). If a function wants to preserve their values across a call, it must save them itself. They aren't just arguments - they can also hold intermediate values between calls.

The following example is deliberately simplified to illustrate the calling convention; real compiler output will look different, but the register assignment principle is the same:

// Simplified illustration of argument passing
MOV     X0, #42        // first integer argument
MOV     X1, #100       // second integer argument
BL      my_function    // call - X0-X7 may be modified
// X0 contains the return value

This convention means that short, simple functions can be very efficient - no need to save and restore registers if you're just doing a quick computation.

X8: indirect results

X8 is not an ordinary ninth argument register. It is specifically used by AAPCS64 to carry the address of caller-allocated storage when a function result is returned indirectly. For example, a result that cannot be returned in the normal result registers may use indirect result return — the caller allocates the result memory and passes its address in X8. There is no requirement for the callee to preserve X8, and X8 is not part of the normal argument sequence.

The caller never passes X8 as an explicit argument in source code - the compiler handles it automatically when the return type requires it:

// Simplified illustration of indirect result
// Caller has allocated storage for the result.
// X8 points to that storage.
BL      get_large_struct
// The result has been written to the caller-provided storage.

X9–X15: caller-clobbered scratch registers

These are caller-clobbered registers available for any intermediate use. The important nuance: "caller-saved" doesn't mean "temporary" in the sense that they can only hold short-lived values. It means the caller cannot assume their values survive a call. A compiler may allocate a long-lived value to X9 if it can prove that no call occurs before the value is consumed - there's nothing in the ABI that prevents it.

// Simplified illustration of scratch register use
ADD     X9, X0, X1     // intermediate result
LSL     X10, X9, #3    // another intermediate
// X9 and X10 are caller-clobbered - a called function may overwrite them

X16–X17: IP0/IP1 and linker scratch

X16 and X17 are known as IP0 and IP1 - intra-procedure-call scratch registers. They have a special relationship with the linker and dynamic loader. The current AAPCS64 explicitly says X16/X17 can be used by call veneers and PLT code.

The important consequence is that X16 and X17 are not safe places to keep values across calls. A linker-generated veneer or PLT sequence may use them even when the source-level callee doesn't. If you're writing hand-written assembly that calls external functions, treat these registers as clobbered.

X18: platform register

X18 is the platform register. Its preservation and usage rules are defined by the platform ABI rather than by the base AAPCS64.

AAPCS64 says that a platform may reserve X18 for interprocedural state such as thread context. If the platform has no such requirement, X18 is an additional caller-saved register. Arm advises that platform-independent developers avoid X18 if possible.

For example, Windows reserves X18 for the Thread Environment Block (TEB), and in user mode it points to the TEB. Android reserves X18 for the Shadow Call Stack (SCS) pointer in hardened binaries - a security mechanism that keeps return addresses on a separate, protected shadow stack. Apple platforms assign their own role to X18 as well. Other platforms have their own rules, so don't assume X18 is freely available without checking the target ABI.

Portable assembly rule: don't use X18 unless you know the target platform ABI.

X19–X28: callee-saved registers

These registers are callee-saved - if a function modifies any of them, it must save their original values and restore them before returning. This is the opposite of the caller-saved convention.

The nuance: "callee-saved" doesn't mean the callee must use these registers. It means if it does use them, it must preserve them. The compiler decides whether to allocate a value to a callee-saved register based on liveness analysis, register pressure, and cost models - not because the ABI mandates it.

If a function keeps a value live across calls, the compiler may choose a callee-saved register. The called function is then required to preserve that register, so the caller doesn't need to save the value around every individual call. The trade-off is that the callee may need to save and restore it in its own prologue and epilogue:

// One possible prologue/epilogue pattern; real compiler-generated
// prologues vary with frame layout, optimization level, unwind
// requirements, and security features.
my_function:
    STP     X29, X30, [SP, #-16]!    // save frame pointer and link register
    STP     X19, X20, [SP, #-16]!    // save callee-saved registers we'll use

    // Because this function modifies X19/X20,
    // it must restore their incoming values before returning.
    MOV     X19, X0                   // preserve first argument across calls
    MOV     X20, X1                   // preserve second argument across calls

    BL      some_other_function       // X19 and X20 survive this call
    // ...

    LDP     X19, X20, [SP], #16      // restore callee-saved registers
    LDP     X29, X30, [SP], #16      // restore frame pointer and link register
    RET

To see why this matters, compare the two paths for a value you need again after a call:

Using caller-saved X9 means you must explicitly save the value before the call and restore it afterward.
Using callee-saved X19 means the responsibility moves into the callee: if the current function itself needs to use X19, it saves it once in its prologue. But any function you call must also preserve X19, so your value survives.

That distinction is the heart of why the ABI has both classes. Values that remain live across calls are often good candidates for callee-saved registers, but the compiler may instead spill, recompute, or otherwise transform the value depending on its cost model.

X29 and X30: frame pointer and link register

X29: frame pointer

X29 is conventionally used as the frame pointer (FP) when a frame pointer is maintained. Optimized code may omit the frame pointer and use X29 as a general-purpose callee-saved register instead - this depends on compiler options (-fno-omit-frame-pointer), debugging and unwinding requirements, and platform rules.

X30: link register

X30 is the link register (LR), and it gets a special role rather than a simple caller/callee classification. When BL (branch with link) executes, the CPU writes the return address into X30 before branching to the target. When the function returns with RET, it jumps to the address in X30.

The practical rule: a function that needs its incoming LR after making another call must preserve that LR. If a function makes another call, that call overwrites X30, so the function must save its incoming return address somewhere else - typically on the stack - if it still needs it to return.

A leaf function that makes no calls can often leave X30 untouched and return directly with RET, avoiding an LR save/restore.

When a frame pointer is used, X29 and X30 are commonly saved together.

Pointer Authentication and X30 security

In modern AArch64 environments (iOS, macOS, and hardened Linux builds), you will often see X30 handled with two additional instructions: PACIASP and AUTIASP. These are part of Armv8.3-A Pointer Authentication (PAC), a hardware security feature designed to prevent Return-Oriented Programming (ROP) and stack-smashing attacks.

Because X30 holds the return address, an attacker who overwrites the saved X30 on the stack can hijack execution when the function returns. PAC solves this by signing the pointer in X30 using a secret key and the current stack pointer (SP) as context before pushing it to memory:

my_function:
    PACIASP                         // Sign X30 using A-key and SP
    STP     X29, X30, [SP, #-16]!   // Push frame pointer and signed X30

    // ... function body ...

    LDP     X29, X30, [SP], #16     // Restore signed X30 from stack
    AUTIASP                         // Authenticate X30 against SP
    RET                             // Return to verified address

PACIASP (prologue): Computes a cryptographic signature from the value in X30 and SP, inserting it into the normally-unused upper bits of X30.
AUTIASP (epilogue): Re-computes the signature using the current SP and validates X30. If the stack was corrupted and the saved X30 was modified, AUTIASP replaces the signature with an invalid address pattern. When RET then attempts to execute, the CPU triggers an immediate translation fault rather than jumping to malicious code.

Crucially for backward compatibility, PACIASP and AUTIASP execute as NOP instructions on legacy Armv8.0 hardware, so binaries built with PAC enabled can still run safely on older AArch64 processors.

SP, XZR, and W registers

SP: stack pointer

SP is a special architectural register rather than an ordinary GPR, and it is not interchangeable with X0–X30. The AAPCS64 requires the stack pointer to be 16-byte aligned at public interfaces, and also requires alignment whenever memory is accessed via SP. AArch64 imposes additional restrictions on how SP can be used by instructions.

Importantly, this 16-byte SP alignment isn't only an ABI convention. The CPU itself enforces an architectural SP Alignment Check whenever SP is used as the base register in a load or store: if SP isn't 16-byte aligned at such an access, the hardware raises an alignment fault regardless of what the ABI says. The ABI rule and the architectural rule work together - the calling convention keeps SP aligned so the hardware check never trips on ordinary code.

AAPCS64 groups SP with the callee-saved registers, but that doesn't mean the callee must restore the exact incoming SP value instruction-for-instruction. The useful rule is: a function must restore SP to the required value before returning, and SP must remain properly aligned.

XZR: the zero register

XZR is the architectural zero register. Reads return zero; writes are discarded. Many instructions use register encoding 31 to mean either SP or XZR, depending on the instruction class: data-processing instructions treat encoding 31 as XZR, while memory instructions and branches treat it as SP.

There is therefore no X31 register you can use like X0–X30; encoding 31 is interpreted as SP or XZR depending on the instruction. This eliminates a common beginner question - there is no architectural X31 general-purpose register.

W registers: the 32-bit view

Each X register also has a 32-bit W-register view. For example, W0 refers to the low 32 bits of X0.

A key AArch64 rule is that writing a W register zeroes the upper 32 bits of the corresponding X register:

MOV     W0, #42
// X0 is now 0x000000000000002A

This is worth remembering when reading compiler output: W0 and X0 aren't separate physical registers. You'll encounter w0, w1, and so on constantly in disassembly.

SIMD and floating-point registers

The V0-V31 registers are 128-bit SIMD/vector registers that also serve as floating-point registers. The calling convention divides them similarly to the general-purpose registers.

Each V register has several aliases depending on the width you're accessing:

V0: Full 128-bit register
Q0: Same 128 bits (quadword)
D0: Low 64 bits (doubleword)
S0: Low 32 bits (singleword)
H0: Low 16 bits (halfword)
B0: Low 8 bits (byte)

V0–V7: argument and return registers

These are the primary SIMD/floating-point argument and return registers. Like their general-purpose counterparts, they're caller-clobbered.

// Simplified illustration: assume S0 and S1 already contain the arguments
BL      my_float_func
// S0 (low 32 bits of V0) contains the return value

V8–V15: partially preserved

Only the low 64 bits of V8–V15 (D8–D15) are callee-saved. That means: if a function modifies the bottom 64 bits of one of these registers, it must restore them before returning; larger values are the caller's responsibility. If you need a full 128-bit value to survive a call, you can't simply put it in V8–V15 and rely on the ABI - you must preserve the upper 64 bits yourself.

V16–V31: temporary registers

These are fully caller-clobbered, like the general-purpose scratch registers. Use them for intermediate SIMD computations that don't need to survive function calls.

Condition flags: NZCV

NZCV is the condition flags register. It holds four single-bit flags:

N (Negative): Set to 1 if the result of the last operation was negative
Z (Zero): Set to 1 if the result was zero
C (Carry): Set to 1 if the last operation produced a carry out
V (Overflow): Set to 1 if the last operation produced a signed overflow

NZCV is modified by flag-setting instructions such as ADDS, SUBS, ANDS, and by comparison instructions such as CMP, CMN, and TST. Ordinary arithmetic instructions such as ADD and SUB do not modify NZCV - you must explicitly use the flag-setting variant.

NZCV is not preserved across a function call; its values are undefined at a public interface. It's also ordinary scratch state within a function: any flag-setting instruction can replace the previous condition flags.

This matters for inline assembly: if your inline assembly modifies condition flags in a way the compiler must not assume is preserved, declare the appropriate "cc" clobber (in GCC/Clang extended inline assembly). Conversely, if your assembly reads condition flags, you need to ensure they've been set appropriately before your block.

// Simplified illustration of NZCV in inline asm
CMP     X0, #0          // flag-setting comparison: sets NZCV
B.NE    is_nonzero       // conditional branch based on Z flag
// ...

What this looks like in compiler output

When you examine compiler output, these conventions explain the register allocation patterns you see. Compilers often use caller-clobbered registers for values that don't need to survive a call, because doing so avoids unnecessary save/restore work. Values that are live across call boundaries may end up in callee-saved registers, but the compiler may instead spill to the stack, recompute the value, or transform the code entirely - register allocation is driven by a cost model, not a simple rule.

Consider a concrete example:

int bar(int);

int foo(int a, int b) {
    int x = a + b;
    return bar(x) + x;
}

x is live across the call to bar() - it's used again afterward. Therefore, the compiler needs some way to preserve its value across the call. It might use a callee-saved register, spill it to the stack, recompute it, or transform the expression. So conceptually:

a, b → x0, x1
x → may move to x19, because x is live across the call to bar()

A complete, valid implementation using a callee-saved register would also have to preserve the caller's incoming X19 in its prologue:

foo:
    STP     X19, X30, [SP, #-16]!     // preserve caller's X19 and our LR

    ADD     W19, W0, W1               // x = a + b, kept in W19
    MOV     W0, W19                   // pass x as the argument
    BL      bar                       // call - x survives in W19
    ADD     W0, W0, W19               // result + x

    LDP     X19, X30, [SP], #16       // restore X19 and LR
    RET

Note what this makes explicit: using a callee-saved register doesn't make the register free. It transfers the preservation responsibility to the current function, which must save and restore X19 around its own body. That's the heart of the distinction between the two classes.

Also notice how the stack is managed: STP X19, X30, [SP, #-16]! pushes two 64-bit registers at once (16 bytes total, hence the #-16 offset), and LDP pops them back together. Storing registers in pairs of 64-bit registers is the standard AArch64 idiom - it's precisely how the code keeps SP moving in 16-byte units, maintaining the required alignment without extra padding. Because the pair takes exactly 16 bytes, the compiler never has to insert alignment holes.

The compiler can also use caller-saved registers for long-lived values when it can prove no call intervenes. "Caller-saved" describes what happens across calls, not how long a value lives.

Writing inline assembly

With GCC/Clang-style extended inline assembly, ABI rules are only part of the contract. You must also tell the compiler which registers, memory, and condition flags your assembly modifies. An instruction sequence can obey the AArch64 ABI and still be invalid inline assembly if its clobbers and operands are incorrectly declared.

Important distinction: in inline assembly the compiler - not your assembly code - is responsible for allocating registers around the asm block. This is fundamentally different from standalone assembly, where you own every register decision.

Specifically:

Describe inputs, outputs, and clobbers using operands and constraints, allowing the compiler to allocate registers unless a specific register is genuinely required.
Use the "cc" clobber if your assembly modifies condition flags.
List any registers your assembly clobbers, or let the compiler allocate them via operand constraints.
Declare memory effects if your assembly accesses memory the compiler doesn't know about.

Don't manually save a caller-saved register in inline asm just because it is caller-saved. Your job is to describe the inputs, outputs, and clobbers accurately. If you need a particular register preserved, model that requirement through operands/constraints rather than silently saving and restoring arbitrary registers.

If your asm modifies a register that the compiler has allocated for something else and you don't tell the compiler, you can break the program even though you didn't violate the ABI. This is the key distinction between standalone assembly and inline assembly embedded in compiler-generated code.

AAPCS64 vs platform ABI

The register conventions described above come from the Arm Procedure Call Standard for AArch64 (AAPCS64), which defines the core calling convention. But AAPCS64 is the base layer - not the whole story. The hierarchy looks like this:

AArch64 architecture
        ↓
    AAPCS64
        ↓
    platform ABI
        ↓
language/runtime/toolchain conventions

Every platform builds on AAPCS64 with its own platform ABI that adds rules specific to the operating system or runtime:

Linux / ELF: AArch64 Linux user-space ABIs generally build on AAPCS64, with additional ELF and platform conventions. The exact ABI also depends on the toolchain and ABI variant being used.
Windows on AArch64: Defines additional ABI rules, including reserving X18 for platform use and requiring specific unwind information.
Apple platforms: Apple platforms build on AAPCS64 with additional platform and toolchain conventions. Apple platforms may also add security-related conventions such as pointer authentication, depending on the target and execution environment. If you're writing hand-written assembly, consult the ABI documentation for the specific Apple target rather than assuming that base AAPCS64 is the entire contract.

Beyond the platform ABI, C++ adds additional ABI requirements beyond plain C argument passing, and things like exception handling, TLS, unwind information, and toolchain-specific conventions can introduce further platform-specific rules.

The practical consequence: when writing hand-written assembly or inline assembly, you need to know not just the AAPCS64 register rules, but also the platform-specific additions. Code that works on one platform may break on another if platform-specific register conventions are violated.

Practical cheat sheet

Here's the compact cheat sheet - the part you'll want to bookmark:

Register	Across a call	Main purpose
X0–X7	❌	Arguments/results
X8	❌	Indirect result address
X9–X15	❌	Scratch
X16–X17	❌	Linker/veneer scratch
X18	Platform-dependent	Platform register or caller-clobbered
X19–X28	✅	Callee-saved
X29	✅	Frame pointer / callee-saved GPR
X30	Special	Link register
SP	✅ (special)	Stack pointer
V0–V7	❌	FP/SIMD args/results
V8–V15	⚠️	Low 64 bits callee-saved
V16–V31	❌	FP/SIMD scratch
NZCV	❌	Condition flags; values are undefined across public interfaces

Conclusion

So what survives a function call?

X19–X29 survive, because the callee must preserve them.
Only the low 64 bits of V8–V15 are guaranteed to survive.
X0–X17 and V0–V7/V16–V31 are generally caller-clobbered.
X30 must be preserved by a non-leaf function if it needs its incoming return address.
NZCV is not something you can carry across a public call - its values are undefined there.

But the deeper lesson is that the ABI describes register obligations, not value lifetimes. The compiler decides where a live value actually goes, choosing between callee-saved registers, spilling, or recomputation based on a cost model. And none of this is enforced by the hardware - it's a contract between separately compiled functions.

Next time you look at compiler output for AArch64, you'll know why certain registers are chosen and what happens to them across function boundaries.

Official documentation and further reading

The ABI specification is a moving target - the current AAPCS64 is the 2025Q4 release, issued in January 2026. Link to the current Arm ABI repository rather than an old developer.arm.com PDF:

Arm ABI repository — AAPCS64 - The authoritative home of the current AAPCS64 and other Arm ABIs
Arm Architecture Reference Manual for AArch64 - The authoritative source for the instruction set and register definitions
Arm System V ABI for AArch64 - The ELF/generic System V conventions that build on AAPCS64
Microsoft ARM64 ABI conventions - Microsoft's platform-specific conventions, including X18 and unwind requirements
Apple Platform ABI for AArch64 - Apple's platform-specific conventions
Linux kernel AArch64 architecture - Platform-specific details for Linux on AArch64

Porting my Rust KVM hypervisor to ARM64: Running the binary

Wed, 22 Jul 2026 18:02:25 +0000

Recap

In the previous article we ported the register handling
of my Rust KVM hypervisor from x86 to ARM64. By tracing QEMU with strace and
inspecting the Linux headers, we discovered that ARM64 accesses CPU state
through the KVM_GET_ONE_REG and KVM_SET_ONE_REG interface and reconstructed
the register id encoding in Rust.

At that point the hypervisor compiled successfully, but trying to access even
the program counter resulted in an ENOEXEC error. In this article we'll
continue the reverse-engineering process, initialize the vCPU correctly, and
finally execute our first ARM64 guest.

Second roadblock: ENOEXEC when getting / setting PC

I was rather optimistic this would work out of the box, but even after trying
to do something as simple as reading the PC register I hit a snag:

ioctl(5<anon_inode:kvm-vcpu:0>, KVM_ARM_SET_DEVICE_ADDR or KVM_GET_ONE_REG, 0x7fdf2c81e8) = -1 ENOEXEC (Exec format error)
Error: Os { code: 8, kind: Uncategorized, message: "Exec format error" }

Luckily, I had a concrete lead now: error code and KVM command. So I searched
for the KVM_GET_ONE_REG in the Linux kernel code on Bootlin Elixir and found
the following code in arch/arm64/kvm/arm.c:

case KVM_SET_ONE_REG:
case KVM_GET_ONE_REG: {
struct kvm_one_reg reg;

r = -ENOEXEC;
if (unlikely(!kvm_vcpu_initialized(vcpu)))
break;

This is exactly the error we were getting and we immediately found what was
missing: vCPU initialization. Actually, if we take a closer look at the code we
will see the comment kernel developers have left us:

int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
{
int err;

/* Force users to call KVM_ARM_VCPU_INIT */
vcpu->arch.target = -1;

Initializing the vCPU

Following the lead I looked for KVM_ARM_VCPU_INIT in the original strace log
and found:

137152 ioctl(12<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57e028) = 0
137156 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7f82f70e68) = 0
137152 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57dd78) = 0
137152 ioctl(10<anon_inode:kvm-vcpu:0>, 0x4020aeae /* KVM_ARM_VCPU_INIT */, 0x7fff57e048) = 0

This wasn't really helpful: we again only see addresses, but we see no hints of
underlying structure. So let's first take a look at how KVM_ARM_VCPU_INIT is
defined in the Linux headers:

$ grep -Rn KVM_ARM_VCPU_INIT /usr/include/
/usr/include/linux/kvm.h:1622:#define KVM_ARM_VCPU_INIT   _IOW(KVMIO,  0xae, struct kvm_vcpu_init)

A quick search for struct kvm_vcpu_init revealed:

$ grep -Rn 'struct kvm_vcpu_init' /usr/include/ -A3
/usr/include/aarch64-linux-gnu/asm/kvm.h:112:struct kvm_vcpu_init {
/usr/include/aarch64-linux-gnu/asm/kvm.h-113-   __u32 target;
/usr/include/aarch64-linux-gnu/asm/kvm.h-114-   __u32 features[7];
/usr/include/aarch64-linux-gnu/asm/kvm.h-115-};

Getting struct data with Python GDB

I didn't want to spend much time understanding every target and feature flag. My
goal was simply to reproduce QEMU's behavior as quickly as possible (and I
didn't even have to understand it completely). Luckily, this structure was
really simple (basically an array of eight 32-bit integers), so I decided to
tackle this with Python GDB automation. I created a breakpoint to stop exactly
on ioctl call when we are in KVM_ARM_VCPU_INIT:

import gdb
import struct

def log(line):
    print(line)
    with open("gdb-debug.out", "a") as f:
        f.write(line + "\n")

class IoctlBreakpoint(gdb.Breakpoint):

    def stop(self):
        if x1 == 0x4020aeae: # KVM_ARM_VCPU_INIT
            ptr = int(gdb.parse_and_eval("$x2"))
            vcpu_init_mem = gdb.selected_inferior().read_memory(ptr, 32) # u32 + 7 * u32
            target_features = struct.unpack("<8I", bytes(vcpu_init_mem))
            log(f"KVM_ARM_VCPU_INIT: target={target_features[0]} features={target_features[1:]}")        
            return False

if __name__ == "__main__":
    tracer = IoctlBreakpoint("ioctl")
    gdb.execute("handle SIGUSR1 SIGUSR2 nostop noprint pass")
    gdb.execute("run")

I ran my QEMU command with GDB:

gdb -ex 'source ./trace-arm.py' ./start-qemu.sh

This has automatically yielded great results:

KVM_ARM_VCPU_INIT: target=5 features=(0, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)
KVM_ARM_VCPU_INIT: target=5 features=(12, 0, 0, 0, 0, 0, 0)

For now I want to try just setting target, but leave features zeroed out. I
could immediately reconstruct this in Rust:

impl VCPU {
    pub fn arm_vcpu_init(&self) -> io::Result<()> {
        let mut vcpu_init : kvm_vcpu_init = unsafe { std::mem::zeroed() };
        // KVM_ARM_VCPU_INIT: target=5 features=(0, 0, 0, 0, 0, 0, 0)
        vcpu_init.target = 5;
        let ret = unsafe { libc::ioctl(self.fd, KVM_ARM_VCPU_INIT, &mut vcpu_init) };
        if ret < 0 {
            return Err(io::Error::last_os_error());
        }
        Ok(())
    }
}

Third roadblock: No KVM exit

Soon after this I was able to get and set registers properly. The first ARM test
binary I compiled was based on working UART code from my hobby operating system
I started on Raspberry Pi. I expected at least a fault exit, if not an MMIO exit
when UART code was called. However, KVM_RUN never returned.

This is where I was stuck for a while. I considered creating a separate thread
that repeatedly queried KVM_GET_ONE_REG for PC, but from my own working
experience I knew that calling KVM ioctl calls asynchronously was a really bad
idea. I also considered adding hardware breakpoint capabilities to at least
trigger KVM_EXIT_DEBUG, but this would require more infrastructure for just
checking simple binary.

Writing a simple binary

I decided to try adding a signal handler instead. If I interrupted KVM_RUN
call with SIGINT maybe I could get registers then synchronously. I lowered
my expectations for initial binary and just created something really simple:

.section .text

.macro curr_el_to reg
mrs\reg, CurrentEL
lsr\reg, \reg, #2
and\reg, \reg, #0xf
.endm

.globl _start
_start:
mrsx0, mpidr_el1
andx0, x0, #0xFF
cbzx0, control
b .

control:
curr_el_to x0
b .

It looks like much, but this code will just load current CPU privilege level to
x0 register (being that level EL=1 is operating-system level, x0 should be
set to 0x1). The mpidr_el1 code is for checking if we are on control CPU
(this should be unnecessary here, but I added this out of paranoia caused by
real-life issues I had on ARM).

Installing signal handler

Finally, I added a signal handler:

extern "C" fn handler(_sig: libc::c_int) { }

unsafe fn install_interrupt_signal() {
    let mut sa: libc::sigaction = std::mem::zeroed();
    sa.sa_sigaction = handler as *const() as usize;
    libc::sigemptyset(&mut sa.sa_mask);
    sa.sa_flags = 0;
    libc::sigaction(libc::SIGINT, &sa, std::ptr::null_mut());
}

The handler itself does nothing. I only needed the signal to interrupt the blocking KVM_RUN call. So my final KVM loop looked like:

    unsafe { install_interrupt_signal() };

    let run = vcpu.kvm_run_mem as *mut kvm_run;

    loop {
        let ret = unsafe { libc::ioctl(vcpu.fd, KVM_RUN, 0usize) };
        if ret < 0 {
            if io::Error::last_os_error().raw_os_error() == Some(libc::EINTR) {
                vcpu.print_regs()?;
            }
            return Err(io::Error::last_os_error());
        }

        let exit_reason = unsafe { (*run).exit_reason };

Running the binary

Finally, it was time to run the gues:

pi@raspberrypi:~ $ ./rust hello-world-arm.img
x0 = 0x0
pc = 0x1000
Loading "hello-world-arm.img" to 0x1000...
^Cx0 = 0x1
pc = 0x101c
Error: Os { code: 4, kind: Interrupted, message: "Interrupted system call" }

We can see that PC did change and that x0 really is set to current CPU
privilege level: 0x1. Why it didn't work for my UART code is probably that the
guest itself requires more initialization than I though (x86 went more smoothly
and felt more plug-and-play).

Conclusion

Although x86 and ARM64 share the same KVM interface at a high level, their
userspace APIs differ significantly. Porting my hypervisor turned out to involve
much more than renaming registers: ARM64 uses per-register ids instead of a
single register structure, requires explicit vCPU initialization through
KVM_ARM_VCPU_INIT and demanded a fair amount of reverse engineering before I
could successfully execute guest code. Fortunately, the same workflow that
worked on x86 like combining strace, GDB and the Linux kernel source proved just
as effective on ARM64 (although required a bit more effort).

The code from this article can be found on my GitHub page:

https://github.com/StjepanPoljak/kvm-rust/tree/kvm-arm-part2-code

The Rise of ARM Servers: Should Developers Care My Take?

Fri, 19 Jun 2026 11:11:54 +0000

You push a build, your CI runner fails on a dependency that only ships prebuilt x86 binaries, and your staging server’s performance bill spikes. You're juggling compatibility headaches, rising cloud costs, and pressure to optimize both speed and energy use all while shipping features.

ARM servers are maturing fast. With major cloud providers offering Arm-based instances, improved tooling (multi-arch containers, cross-compilers), and real-world wins in cost and efficiency, ARM is no longer an academic alternative, it's a practical option. This post walks through what ARM servers mean for developers, when to care, and concrete steps to evaluate or migrate your workloads.

Why ARM servers are suddenly everywhere

ARM (Advanced RISC Machines) is a CPU architecture that emphasizes efficiency and a simpler instruction set.
Historically dominant in mobile and embedded devices (think smartphones), ARM is now competitive in the server space thanks to:
- Custom cloud silicon (e.g., AWS Graviton, Ampere Altra) delivering good price/performance.
- Broader OS and tooling support (Linux distros, container runtimes, package managers).
- Developer-friendly ecosystems: languages (Go, Rust), runtimes (Node, Java, Python), and containerization workflows increasingly support arm64.
- Power efficiency and density advantages for data centers and edge deployments.

Should developers care? Short answer: Yes — but selectively.

Not every team needs to rush into ARM. But every developer should care enough to understand the trade-offs and be able to evaluate the benefits for their stack.

Key areas where ARM matters:

Cost and price performance: For many cloud workloads, arm64 instances can be cheaper for the same throughput.
Energy and sustainability: ARM’s efficiency lowers power draw relevant for green initiatives and edge deployments.
Edge and IoT convergence: If you deploy to ARM-based edge devices, aligning server-side architecture reduces friction.
Performance: Some integer-heavy server workloads perform very well on modern ARM CPUs; others (specialized AVX-heavy workloads) may still favor x86.
Ecosystem maturity: Tooling has improved dramatically, multi-arch images, CI support, and native packages are common now.

When ARM might not be a fit:

Heavy reliance on x86-only native binaries or proprietary drivers.
Applications tightly coupled to SIMD/AVX-512 optimizations that aren’t available on ARM.
Teams with large legacy stacks where migration cost outweighs operational savings.

Practical checklist: Should your team pilot ARM servers?

Use this quick checklist to decide whether to test ARM for your workloads:

Is your app primarily high-level language code (Go, Rust, Node.js, Python, Java)? → Good candidate.
Do you use lots of native C/C++ extensions with prebuilt x86-only binaries? → Investigate compatibility or recompilation paths.
Are you looking to reduce cloud costs or improve energy efficiency? → Worth testing.
Do you deploy to edge or ARM-based devices? → Strong alignment benefit.
Can your CI/CD pipeline build multi-arch images and run arm64 tests? → Necessary for safe rollout.

If you checked two or more “yes” answers, consider a small pilot (e.g., one microservice or a staging environment).

How to evaluate ARM — a practical approach

Start small
- Pick a stateless microservice or a low-risk backend job.
- Run benchmarks on both x86 and arm64 instances for representative workloads.

Use multi-arch containers

Build multi-architecture images to support both x86_64 and arm64:

# Using Docker buildx (example)
docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 -t myorg/myapp:latest --push .

Run tests natively on arm64
- Avoid relying solely on emulation for performance tests. Launch a small arm64 instance in your cloud provider or use a local ARM machine.
Check dependencies
- Audit native modules and third-party binaries. For Node.js, look for native addons; for Python, check wheels availability for arm64.
- If native code exists, plan to recompile or replace with pure-language alternatives.
CI/CD updates
- Add arm64 runners or use cloud-based arm64 CI builders.
- Use cross-compilation for compiled languages where possible.
Measure and iterate
- Collect metrics: latency, throughput, CPU utilization, memory use, and cost.
- Compare price/performance, not just raw CPU numbers.

Concrete migration tips and commands

Detect architecture:

uname -m   # returns aarch64 for ARM64; x86_64 for x86

Go: cross-compile for Linux/ARM64

GOOS=linux GOARCH=arm64 go build -o myapp-linux-arm64 ./cmd/myapp

Node.js: rebuild native modules on arm64

npm rebuild --arch=arm64
# or build multi-arch images and run `npm install` inside arm64 build step

Python: prefer manylinux wheels or build from source in arm64 environment. Use pip with platform-specific wheels if available.
Emulation (use sparingly for tests, not production):
- Docker Buildx can use QEMU for cross-building, but native testing on arm64 instances will be more reliable.

Performance and compatibility considerations

Performance varies by workload:
- Compute-bound tasks may do very well on modern ARM, but some vectorized workloads (heavily tuned for AVX) might still favor x86.
Library support:
- Popular libraries now ship arm64 builds, but niche or enterprise packages may lag.
Tooling:
- Container runtimes, orchestrators (Kubernetes), and observability stacks have arm64 support — though some vendor plugins may lag.

DevOps and CI implications

Build pipelines should produce and test multi-arch artifacts.
CI runners: add arm64 runners (self-hosted or cloud-provided) to avoid blind spots.
Monitoring: ensure observability and APM tools can instrument arm64 instances.
Infrastructure as Code: update images and instance types in IaC templates (Terraform, CloudFormation).

Keywords: arm64 CI runners, Kubernetes arm64, IaC for ARM

Example migration plan (minimal risk)

Choose a candidate service (stateless, low-calls).
Build multi-arch container images.
Deploy to a small arm64 staging environment.
Run full integration and load tests.
Compare metrics and costs over a week under representative load.
If results are positive, roll out more services in phases.

Common pitfalls to avoid

Assuming everything “just works", always test native dependencies.
Skipping production-like tests, emulation can hide problems.
Not updating CI, builds that pass on x86 may fail on arm64 at deployment time.

Final verdict for developers

ARM servers are no longer niche. For many modern applications, especially those written in portable languages and deployed in the cloud —> arm64 offers tangible benefits in cost and energy efficiency. Developers should:

Understand the architecture differences and test key workloads.
Adopt multi-arch build pipelines and arm64 CI runners.
Audit dependencies and plan recompilation or replacement where needed.

If you maintain services that are performance sensitive or rely on specialized x86 optimizations, don’t rush but do prepare. At minimum, integrate arm64 into your testing matrix so you aren’t surprised when your cloud provider offers a cheaper or faster option.

Call to action

Tried ARM in your stack? Share your wins, gotchas, or benchmark numbers in the comments — it helps other developers make better decisions. If you found this useful, share the post with your team and bookmark the page for migration checklists and follow-ups. Want a migration checklist tailored to your app? Reply with your stack details (language, dependencies, CI) and I’ll outline a concrete pilot plan.