Coder Legion - Recent posts tagged arm64

How to Fix an ARM64 WAF Deploy When Personal Edition Is Not Supported

Sun, 01 Nov 2026 03:46:59 +0000

An ARM64 host can meet every published dependency and still be configured as an unsupported SafeLine deployment. The concrete misconfiguration is a manual .env left on the sample values: ARCH_SUFFIX is empty, POSTGRES_IMAGE is chaitin/safeline-postgres:15.18, and the operator plans to run Personal Edition. The Deploy guide says an ARM server must set ARCH_SUFFIX=-arm and POSTGRES_IMAGE=chaitin/safeline-postgres-arm:15.18. It also states that SafeLine installed on ARM requires a Pro license to work properly, and that the Personal Edition is not supported on ARM (https://docs.waf.chaitin.com/en/GetStarted/Deploy).

Confirm the host before editing .env

SafeLine lists Linux, instruction architectures x86_64 and arm64, Docker 20.10.14 or above, Docker Compose 2.0.0 or above, and a minimum environment of 1 core CPU, 1 GB memory, and 5 GB disk (https://docs.waf.chaitin.com/en/GetStarted/Deploy). The same page gives the checks: uname -m for instruction architecture, docker version, docker compose version, free -h, and df -h. It also lists a CPU ssse3 check with lscpu | grep ssse3. Run uname -m first. If it reports an ARM architecture, do not keep the blank ARCH_SUFFIX from the sample .env. A blank suffix is the sample default, not the ARM setting.

The Deploy guide also offers an automatic install command that requires root. This fix follows the manual path, because that is where ARCH_SUFFIX and POSTGRES_IMAGE are written by hand. Create the data directory with mkdir -p "/data/safeline", download https://waf.chaitin.com/release/latest/compose.yaml into that directory, and write .env there. The sample sets SAFELINE_DIR=/data/safeline, IMAGE_TAG=latest, MGT_PORT=9443, a Postgres password, SUBNET_PREFIX=172.22.222, IMAGE_PREFIX=chaitin, an empty ARCH_SUFFIX, an empty RELEASE, REGION=-g, and MGT_PROXY=0. On ARM, replace the architecture lines before the first launch. Set ARCH_SUFFIX=-arm and POSTGRES_IMAGE=chaitin/safeline-postgres-arm:15.18. Leave RELEASE empty unless you intentionally want the LTS build. The Deploy guide says the LTS version will no longer be maintained starting from 9.1.0-LTS (14 August 2025) and strongly recommends an upgrade to the latest version.

Generate the documented secrets in the SafeLine directory, write DETECTOR_MGMT_TOKEN, MGT_INTERNAL_TOKEN, and DETECTOR_AES_KEY into .env, and chmod 600 .env. Those steps are separate from the architecture mistake, but a correct ARM file still needs them before docker compose up -d. Also create the documented resource directories and apply the documented permissions before launch, including chmod 700 on the run, mgt, sock, and postgres data paths named in the guide, plus the documented socket link.

The edition limit is not an image-tag typo

Changing the suffix does not remove the edition limit. The Deploy guide is explicit: SafeLine installed on ARM requires a Pro license to work properly, and the Personal Edition is not supported on ARM. If the only edition you can activate is Personal Edition, correcting ARCH_SUFFIX is not enough to make that edition supported. Treat a Personal Edition plan on this host as part of the same misconfiguration, not as a tuning choice. The supported path in that document is a Pro license together with the ARM image settings.

The Deploy guide does not publish a sample denial log for a Personal Edition install on ARM. Do not treat a guessed error string as proof. Use the written requirement as the decision, then use the console screen below as proof that the corrected deploy is actually up.

The screen that confirms the fix

After ARCH_SUFFIX=-arm, the ARM Postgres image line, and a Pro license are in place, start the stack with docker compose up -d from /data/safeline. The Deploy guide says this can take several minutes. The confirmation is the web console the guide tells you to open. Browse to https://<safeline-ip>:9443/ and you should see the web console page. Then run docker exec safeline-mgt resetadmin. A successful run prints the initial username, the initial password, and a Done line. Keep that password and log in. That console page, together with those resetadmin lines, is the screen that confirms the architecture settings were applied and the service reached the documented ready state.

If the console URL does not load, re-read .env before changing anything else. The two lines that define this misconfiguration are still the ones to check: ARCH_SUFFIX must be -arm, and POSTGRES_IMAGE must be chaitin/safeline-postgres-arm:15.18. Also confirm MGT_PORT is still 9443 unless you chose another console port, because the documented URL uses that port. MGT_PROXY should stay 0 unless you are configuring a proxy for the console. Re-check uname -m if you are no longer sure the host is ARM. The architecture check and the .env lines have to agree before the console can be treated as a fixed ARM deploy.

After the console is up

SafeLine is a self-hosted WAF placed in front of a web application. The product introduction says it filters and monitors HTTP traffic and can defend against SQL injection, XSS, code injection, OS command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP flood, and bot abuse (https://docs.waf.chaitin.com/en/home). Core capabilities listed there include defenses for web attacks, proactive bot abused defense, HTML and JS code encryption, IP-based rate limiting, and a web access control list. The docs also state that SafeLine is production-ready. It works as a reverse proxy: clients pass through SafeLine before they reach the server, and SafeLine forwards traffic that passes policy (https://docs.waf.chaitin.com/en/GetStarted/AddApplication).

Once login works, open the Applications page and click Add Application. Enter the domain, hostname, or IP, the listen port such as 80 or 443, check SSL for an HTTPS app, and set the upstream to the real address of the original app. Resolve that domain to the IP of the server where SafeLine runs. The configuration guide shows sample requests for SQL injection, XSS, path traversal, code injection, and XXE, and says a blocking page means the attack was defended. Open the console attack list and click detail for the event. That check proves protection on a live site. It does not replace the resetadmin screen that confirmed the ARM deploy itself.

Does Personal Edition run on an ARM64 SafeLine host?

No. The Deploy guide says SafeLine installed on ARM requires a Pro license to work properly, and the Personal Edition is not supported on ARM (https://docs.waf.chaitin.com/en/GetStarted/Deploy).

Which environment lines are wrong on ARM if I keep the sample file?

ARCH_SUFFIX is empty in the sample, and the sample Postgres image is chaitin/safeline-postgres:15.18. On an ARM server those must become ARCH_SUFFIX=-arm and POSTGRES_IMAGE=chaitin/safeline-postgres-arm:15.18.

What output confirms the corrected deploy?

The web console at https://<safeline-ip>:9443/ should load, and docker exec safeline-mgt resetadmin should print the initial username, the initial password, and Done.

Is arm64 outside the published architecture list?

No. The dependency table lists instruction architectures x86_64 and arm64. The command uname -m is the documented way to view the instruction architecture before you choose ARCH_SUFFIX.

What should I verify before the first docker compose up?

Confirm Docker is 20.10.14 or above, Docker Compose is 2.0.0 or above, and the host has at least 1 core, 1 GB memory, and 5 GB disk. On ARM, also confirm the two image lines and that you are not relying on Personal Edition.

SafeLine has surpassed 700,000 installations worldwide as of September 30, 2026. SafeLine Pro annual promotional pricing is $650-$880, compared with the regular $1,000 per year.

SafeLine website: https://ly.cyberserval.tech/MqhYdmI
Demo: https://demo.waf.chaitin.com:9443/statistics
Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Pro offer: https://cyberserval.tech/landing/lto

Why I Check ARM64 Edition Limits Before Turning SafeLine On

Sun, 01 Nov 2026 03:46:28 +0000

I do not turn protection on until an ARM64 host has passed an edition check. Architecture support and edition support are different facts. The deploy guide lists both x86_64 and arm64, then states that SafeLine installed on ARM requires a Pro license to work properly and that the Personal Edition is not supported on ARM (https://docs.waf.chaitin.com/en/GetStarted/Deploy). If I treat arm64 as permission to use Personal Edition, I can finish Docker setup and still have an instance that is not allowed to do the job.

This is the checklist I run before any public cutover.

Confirm the edition before touching images

I run uname -m first, because that is the documented way to view instruction architecture. If the host is not x86_64, I treat it as an ARM deployment and stop unless a Pro license is already available. I do not infer a price, a trial, or an activation command. The deploy page states the requirement and does not document those extras, so they stay outside the decision.

Image settings come next, and only after the license gate. On an ARM server the same guide says to set ARCH_SUFFIX=-arm and POSTGRES_IMAGE=chaitin/safeline-postgres-arm:15.18. Those values select ARM builds. They do not waive the Personal Edition limit. My written rule is simple: ARM images without a confirmed Pro license means do not install, or uninstall if a test install already happened. The documented removal path is docker compose down in the SafeLine directory, then removal of that directory only when I am sure the data can go.

Prove the host meets the published minimum

Support for arm64 does not mean every small board is a safe place to put a reverse proxy. The deploy guide requires Linux, Docker 20.10.14 or above, and Docker Compose 2.0.0 or above. The minimum environment is 1 CPU core, 1 GB of memory, and 5 GB of disk. The CPU must provide ssse3. I use the listed checks: docker version, docker compose version, free -h, df -h, processor lines from /proc/cpuinfo, and lscpu | grep ssse3.

I also reserve the data directory before install. SafeLine writes configuration and data there, and the guide expects at least 5 GB. The manual layout uses /data/safeline. If memory, disk, Docker, or ssse3 fails, I fix the host first. Putting a filter in front of a healthy origin on an unfit host creates an outage that looks like an attack.

Decide what on will change for clients

SafeLine is a self-hosted web application firewall. It sits in front of an application as a reverse proxy, filters and monitors HTTP and HTTPS, and blocks malicious traffic while forwarding the rest (https://docs.waf.chaitin.com/en/home). The application guide adds that it is developed based on nginx: it receives traffic for the original site, cleans malicious traffic, and forwards safe traffic upstream (https://docs.waf.chaitin.com/en/GetStarted/AddApplication).

Before I add an application, I write down the three required fields. Domain can be a domain name, a hostname, or an IP address. Port is what SafeLine will listen on, such as 80 or 443, with SSL checked for HTTPS applications. Upstream is the real address SafeLine will use to reach the origin. After those settings, the domain has to resolve to the SafeLine server. That DNS change is the moment protection is actually on for visitors. I do not make it until the upstream answers through the proxy.

I also pick controls deliberately instead of enabling every feature on the first day. Documented capabilities include defenses against SQL injection, XSS, code injection, OS command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP flood, and bot abuse. Separate controls cover IP-based rate limiting, a web access control list, CAPTCHA challenge, an authentication challenge that blocks visitors who do not enter the password, and dynamic protection that encrypts HTML and JavaScript on each visit. Rate limiting is described as throttling traffic that exceeds defined limits, including DoS attempts, brute-force attempts, and traffic surges. I enable the extra challenges only after a normal page load works. A password wall or a CAPTCHA on a broken upstream hides the real fault.

Choose the install path and the release channel

Automatic install is the recommended path. The English command in the deploy guide is bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en, and it requires root. I still run the edition and host gates before I paste it, because a successful script is not the same as a supported Personal Edition on ARM.

Manual install is the path I use when I want the ARM suffix visible before docker compose up -d. That path installs Docker if needed, creates /data/safeline, downloads compose.yaml, writes .env, generates the documented secrets, and fixes permissions. The console port is MGT_PORT, 9443 unless I change it. After startup I open https://<safeline-ip>:9443/ and create the first administrator with docker exec safeline-mgt resetadmin. I store that initial password before anyone else logs in.

RELEASE=-lts is a separate decision. The deploy guide says the LTS version will no longer be maintained starting from 9.1.0-LTS on 14 August 2025, and it strongly recommends upgrading to the latest version. If I am uneasy about a new release, the same page recommends upgrading about one month later. I never choose LTS as a way around the ARM Pro requirement.

Turn the proxy on, then test, then cut over

When the checks above are true, I add the application and only then move a test name to the SafeLine address. I replay the documented probes against my own domain: a SQL injection union-select, an XSS onerror payload, a path traversal toward /etc/passwd, a code-injection probe, and an XXE-style payload. A blocking page means that request was stopped. I open the console attack list and read the detail view so the block is recorded, not just displayed. I also load a known-good URL. Protection is on only when bad requests are blocked, good requests still reach the origin, and I can move DNS back if the proxy path fails.

Does Personal Edition work on ARM64?

No. The deploy guide says SafeLine on ARM requires a Pro license to work properly, and that the Personal Edition is not supported on ARM. ARCH_SUFFIX=-arm only selects ARM images.

What host checks should pass before installation?

Linux on x86_64 or arm64, Docker 20.10.14 or newer, Docker Compose 2.0.0 or newer, at least 1 core, 1 GB of memory, 5 GB of disk, and ssse3. The deploy guide names uname -m, the Docker version commands, free -h, df -h, CPU info, and lscpu | grep ssse3.

What changes when protection is turned on?

Visitors reach SafeLine instead of the origin. SafeLine listens on the configured port, filters HTTP or HTTPS, and forwards safe traffic to the upstream. The application domain must resolve to the SafeLine server, so the cutover is a DNS and proxy change.

How can I test blocking without inventing new attacks?

Use the probes in the application guide, pointed at your own domain: SQL injection, XSS, path traversal, code injection, and XXE. Expect a blocking page, then confirm the event in the console attack list detail view. Also request a normal page so you know the upstream still works.

Does the LTS channel remove the ARM edition limit?

No. LTS is a release choice, set with RELEASE=-lts. Maintenance for that line stops at 9.1.0-LTS, dated 14 August 2025, and the deploy guide recommends the latest version. The Pro license requirement for ARM remains either way.

SafeLine has surpassed 700,000 installations worldwide as of September 30, 2026. SafeLine Pro annual promotional pricing is $650-$880, compared with the regular $1,000 per year.

SafeLine website: https://ly.cyberserval.tech/MqhYdmI
Demo: https://demo.waf.chaitin.com:9443/statistics
Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Pro offer: https://cyberserval.tech/landing/lto

Booting ARM64 Linux in my Rust KVM Hypervisor: Getting to the console with help of eBPF

Fri, 25 Sep 2026 14:40:06 +0000

Recap

In the previous article I got ARM64 Linux as far
as executing inside my Rust KVM hypervisor. I still couldn't actually interact
with the guest, though: I had no working console.

This article is about getting from "the PC is moving" to typing "hello" into
Linux and getting "HELLO" back.

UART

Last time we learned how to obtain the device tree blob from QEMU's virt
machine. Now we want to take a look at what's inside. We can decompile it:

dtc -I dtb -O dts -o virt.dts virt.dtb

Then, a simple search through the file already gives us the clue we need:

pi@raspberrypi:~ $ grep -i uart virt.dts -A5 -B1
        pl011@9000000 {
                clock-names = "uartclk\0apb_pclk";
                clocks = <0x8000 0x8000>;
                interrupts = <0x00 0x01 0x04>;
                reg = <0x00 0x9000000 0x00 0x1000>;
                compatible = "arm,pl011\0arm,primecell";
};

So, now I had a lead: I needed to emulate the pl011 UART at 0x09000000 so
that the existing Linux pl011 driver could communicate with it. This was not a
really interesting part, as most of it was simply reading Linux kernel source
code along with pl011 documentation from ARM. Soon I was able to get output
from Linux and it felt really great:

[    0.000000] Linux version 7.1.3 (docker@stjepan-Aspire-F5-573G) (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT Thu Jul 16 08:52:24 UTC 2026
[    0.000000] KASLR disabled on command line
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: UEFI not found.
[    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x0000000047ffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x47fc10c0-0x47fc37bf]
                  ### omitted further kernel log ###

Creating custom initramfs

Now I needed bidirectional UART support so that I could send input to the guest
as well. To make further investigation easier, I created a very simple binary
written in C to be run as the Linux init process: it just prints out current
interrupt statistics and asks user for input, echoing it back in all caps. All
we then need is initramfs.txt file with some simple folders and devices:

dir / 0755 0 0
dir /dev 0755 0 0
dir /proc 0555 0 0
dir /sys 0555 0 0
nod /dev/kmsg 0600 0 0 c 1 11
nod /dev/console 0600 0 0 c 5 1
file /init /home/stjepanp/Develop/initramfs/init 0755 0 0

I only needed enough of a filesystem to provide /init, /dev/console, and
access to /proc for interrupt statistics. Then, in the Linux source code
folder, we generate initramfs with:

./usr/gen_initramfs.sh ../initramfs/initramfs.txt -o initramfs

To see at which memory address to add initramfs, we can run ordinary QEMU with
our initramfs and see what happens:

KERNEL=/home/pi/Image
qemu-system-aarch64                                             \
        -M virt,dumpdtb=virt.dtb                                \
        -enable-kvm                                             \
        -cpu host                                               \
        -kernel ${KERNEL}                                       \
        -append "console=ttyAMA0 nokaslr rdinit=/init"          \
        -serial stdio                                           \
        -nographic                                              \
        -nodefaults                                             \
        -initrd "./initramfs"

After decompiling virt.dtb (explained in previous article) we can see the location:

pi@raspberrypi:~/dtb $ grep initrd virt-initramfs.dts -B2 -A3
        chosen {
                bootargs = "console=ttyAMA0 initramfs_async=0 rdinit=/init nokaslr";
                linux,initrd-start = <0x0 0x40200000>;
                linux,initrd-end = <0x0 0x40294200>;
                stdout-path = "/pl011@9000000";
                kaslr-seed = <0x8114b902 0xe815aa93>;
        };

We do have to recalculate initrd-end if size of our initramfs file changes.
If that happens, we can easily recompile the device tree blob after changing
the value. Also, let's not forget to load initramfs in Rust code:

const LOAD_ADDR: u64 = 0x40_000_000;
const INITRAMFS_OFFS: u64 = 0x200_000;
const KERN_OFFS: u64 = 0x200_000 + INITRAMFS_OFFS;

impl VM {
    pub fn arch_load_linux(&mut self, vcpu: &mut VCPU, args: &Args) -> io::Result<()> {
        let linux_mem_idx = self.add_mem_region(1024 * 1024 * 1024, LOAD_ADDR)?;
        self.load_file_to_memory(linux_mem_idx, &dtb_path, 0x0)?;
        self.load_file_to_memory(linux_mem_idx, &initramfs_path, INITRAMFS_OFFS)?;
// ...
    }

Setting up vGIC

Even without having the ability to send keystrokes there was one big problem
with the Linux console stdout: The kernel messages were reaching the console,
but once execution reached /init, output stopped behaving as expected. It
took me some time to realize that the ARM architectural timer was also involved.
Its interrupts need to be delivered through the virtual GIC, so I first needed
to create a vGIC. It turned out creating the vGIC was easy. I just followed the
logic from QEMU source code:

https://elixir.bootlin.com/qemu/v11.0.1/source/accel/kvm/kvm-all.c#L4035

In Rust this translates to:

let mut create_device: kvm_create_device = unsafe { std::mem::zeroed() };
create_device.type_ = KVM_ARM_DEVICE_VGIC_V2;
create_device.fd = 0xffffffff;
create_device.flags = 0;

unsafe { libc::ioctl(self.fd, KVM_CREATE_DEVICE, &mut create_device); }

This worked. Also note that I have omitted proper error handling to keep the
code short and concise.

Tracing IRQ injection

Now if we run our initramfs with strace as before, and we try to type in
some keys, we will see that in our log we have:

pi@raspberrypi:~ $ grep -i irq kvm.log | tail -n5
458398 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7fd932dbd0) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db30) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db20) = 0
458403 ioctl(9<anon_inode:kvm-vm>, 0x4008ae61 /* KVM_IRQ_LINE */, 0x7f85c1db70) = 0

So our ioctl for IRQ injection actually takes an address as argument. If we
investigate this, we can find out how KVM_IRQ_LINE is handled in the Linux
kernel:

https://elixir.bootlin.com/linux/v6.1.21/source/virt/kvm/kvm_main.c#L4776

Reading this code we see that the relevant function is kvm_vm_ioctl_irq_line
that takes struct kvm_irq_level as second argument and this is what we need
to interpret:

/* for KVM_IRQ_LINE */
struct kvm_irq_level {
/*
 * ACPI gsi notion of irq.
 * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47..
 * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23..
 * For ARM: See Documentation/virt/kvm/api.rst
 */
union {
__u32 irq;
__s32 status;
};
__u32 level;
};

For this purpose, we can treat it as an 8-byte structure containing the IRQ number and level. Therefore, we can easily inspect it again via eBPF when we
run QEMU with our initramfs:

pi@raspberrypi:~ $ sudo bpftrace -e 'kprobe:kvm_vm_ioctl_irq_line { printf("irq_level=%x\n", *((uint64 *)arg1)); }'
Attaching 1 probe...
irq_level=1000021
irq_level=1000021
irq_level=1000021
irq_level=1000021

Without wasting much time we can easily reconstruct this behavior in Rust:

pub fn arch_update_irq(level: u32, vm_fd: libc::c_int) -> io::Result<()> {
    let mut irq: kvm_irq_level = unsafe { std::mem::zeroed() };
    irq.level = level;
    irq.__bindgen_anon_1.irq = 0x1000021;

    let ret = unsafe {
        libc::ioctl(vm_fd, KVM_IRQ_LINE, &mut irq)
    };

    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    Ok(())
}

Note that, in essence, 0x1000021 is not an arbitrary magic number. On ARM64,
KVM_IRQ_LINE encodes the interrupt type and interrupt ID in the irq field. Here 0x01 selects an SPI and 0x21 is interrupt 33 the UART interrupt from
the device tree.

No such device or address

However, when we run our program, it fails with "No such device or address". To
get some clue on what is going on, we can try strace and get:

ioctl(5, KVM_RUN, 0)                    = -1 ENXIO (No such device or address)
Error: Os { code: 6, kind: Uncategorized, message: "No such device or address" }

My first impulse was to search for ENXIO in Elixir bootlin, but,
unfortunately, there were simply too many functions and branches that could
return ENXIO. Rather than trying to follow every possible ENXIO path
manually, I decided to use a more brute-force approach. I attached kretprobes
to the KVM functions and looked for one returning -ENXIO:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:kvm_* /retval == (uint32)(-6)/ { printf("%s\n", probe); }'
Attaching 487 probes...
kretprobe:kvm_vgic_map_resources
kretprobe:kvm_arch_vcpu_run_pid_change

The second function here does not seem really relevant, so I tried looking for
kvm_vgic_map_resources on Elixir:

https://elixir.bootlin.com/linux/v6.1.21/source/arch/arm64/kvm/vgic/vgic-v2.c#L289

Now taking a look at the function we can clearly see where ENXIO is returned:

int vgic_v2_map_resources(struct kvm *kvm)
{
    struct vgic_dist *dist = &kvm->arch.vgic;
    int ret = 0;

    if (IS_VGIC_ADDR_UNDEF(dist->vgic_dist_base) ||
        IS_VGIC_ADDR_UNDEF(dist->vgic_cpu_base)) {
        kvm_debug("Need to set vgic cpu and dist addresses first\n");
        return -ENXIO;
    }
    // ...

Now we have a hint. We need to set vGIC CPU and distributor addresses first. To
confirm our hypothesis we can try tracing this function specifically:

pi@raspberrypi:~ $ sudo bpftrace -e 'kretprobe:vgic_v2_map_resources { printf("ret=%u\n", retval); }'
Attaching 1 probe...
ret=4294967290

Finding out distributor address

My first attempt to set up distributor address didn't go well. I wrote the
following code (I extracted distributor address from the device tree and
tried to reconstruct the logic from QEMU source code):

let dist_addr: u64 = 0x08_000_000;
let mut dev_attr: kvm_device_attr = unsafe { std::mem::zeroed() };
dev_attr.group = KVM_DEV_ARM_VGIC_GRP_ADDR;
dev_attr.attr = KVM_VGIC_V2_ADDR_TYPE_DIST;
dev_attr.flags = 0;
dev_attr.addr = dist_addr;

unsafe { libc::ioctl(create_device.fd as i32, KVM_SET_DEVICE_ATTR, &mut dev_attr) };

However, when I ran the program, I got:

ioctl(6, KVM_SET_DEVICE_ATTR, 0x7ffe2696d8) = -1 EFAULT (Bad address)

To debug this, I needed to inspect the values defined in
struct kvm_device_attr, but it was a lot more complicated: I had to check if
the struct I was tracing was for KVM_DEV_ARM_VGIC_GRP_ADDR group and for
KVM_VGIC_V2_ADDR_TYPE_DIST. So, I decided to write my eBPF program in C with
and called it probe-kvm.bpf.c:

#include <linux/bpf.h>
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <linux/kvm.h>
#include <stdbool.h>
#include <stdint.h>

char LICENSE[] SEC("license") = "GPL";

SEC("kprobe/vgic_v2_set_attr")
int BPF_KPROBE(kprobe_vgic_v2_set_attr, void *dev, struct kvm_device_attr *dev_attr)
{
        struct kvm_device_attr attr;
        __u64 value;

        bpf_probe_read_kernel(&attr, sizeof(attr), dev_attr);
        if (attr.group != KVM_DEV_ARM_VGIC_GRP_ADDR) {
                return 0;
        }

        if ((attr.attr != KVM_VGIC_V2_ADDR_TYPE_DIST) && (attr.attr != KVM_VGIC_V2_ADDR_TYPE_CPU)) {
                return 0;
        }

        bpf_printk("\nset_attr: attr=%s addr=0x%llx", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr);

        return 0;
}

To compile and run it I wrote the following script:

#!/bin/sh

clang -O2 -g -mcpu=v1 -target bpf -I /usr/include/aarch64-linux-gnu -c probe-kvm.bpf.c -D__TARGET_ARCH_arm64 -o probe-kvm.bpf.o
sudo rm -rf /sys/fs/bpf/vgic
sudo bpftool prog loadall probe-kvm.bpf.o /sys/fs/bpf/vgic autoattach
sudo cat /sys/kernel/debug/tracing/trace_pipe

So, I started ordinary QEMU along my eBPF trace and got this output:

$ ./start-probe.sh
           <...>-146994  [002] d..31 953951.139387: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fd4d3c450

           <...>-146994  [002] d..31 953951.139418: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fd4d3c450

This really looked like a userspace address while I was supplying a raw address.
To find out what was at that address I modified the last line of the BPF program
above:

        if (bpf_probe_read_user(&value, sizeof(value), (void *)attr.addr) != 0)
                return -1;

        bpf_printk("\nset_attr: attr=%s addr=0x%llx, *addr=0x%llx\n", attr.attr == KVM_VGIC_V2_ADDR_TYPE_DIST ? "DIST" : "CPU", attr.addr, value);

With this I got what I didn't really expect:

           <...>-147143  [000] d..31 954397.183145: bpf_trace_printk: 
set_attr: attr=CPU addr=0x7fdc806ca0, *addr=0x8010000

           <...>-147143  [000] d..31 954397.183177: bpf_trace_printk: 
set_attr: attr=DIST addr=0x7fdc806ca0, *addr=0x8000000

So the addr field is actually a pointer to the address, and not the address
itself. Note that 0x8010000 and 0x8000000 are the guest physical
addresses of the GIC CPU interface and distributor taken from the virt machine's
device tree:

pi@raspberrypi:~/dtb $ grep -i intc virt.dts -A3
        intc@8000000 {
                phandle = <0x8001>;
                reg = <0x00 0x8000000 0x00 0x10000 0x00 0x8010000 0x00 0x10000>;
                compatible = "arm,cortex-a15-gic";

This also explained the EFAULT: KVM was trying to dereference
attr.addr as a userspace pointer. With this small fix, I got a fully working
Linux console:

dev_attr.addr = &dist_addr as *const u64 as u64;

The following snippet shows that my init process (as part of
initramfs) was run and it echoed back my input in capital letters:

[    0.832623] clk: Disabling unused clocks
[    0.834046] PM: genpd: Disabling unused power domains
[    0.845565] Freeing unused kernel memory: 12672K
[    0.847475] Run /init as init process
           CPU0       
 11:        199 GIC-0  27 Level     arch_timer
 13:          0 GIC-0  33 Level     uart-pl011
IPI0:         0       Rescheduling interrupts
IPI1:         0       Function call interrupts
IPI2:         0       CPU stop interrupts
IPI3:         0       CPU stop NMIs
IPI4:         0       Timer broadcast interrupts
IPI5:       168       IRQ work interrupts
IPI6:         0       CPU backtrace interrupts
IPI7:         0       KGDB roundup interrupts
Err:          0

type a string: hello
allcaps: HELLO

[    2.224776] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    2.227382] CPU: 0 UID: 0 PID: 1 Comm: init Not tainted 7.1.3 #1 PREEMPT 
[    2.229749] Hardware name: linux,dummy-virt (DT)

The interrupt statistics are also printed from my init. Also, when init
exits, we get the (correct) Kernel panic that init process was terminated.

Final thoughts

At this point I finally had what I was looking for: a real Linux userspace
running inside my own KVM-based hypervisor, with a working UART, interrupt
delivery and a minimal initramfs.

More importantly, I now had a way to interact with the guest and inspect what
was happening inside KVM. That gives me a much better foundation for the next
step: implementing more of the virtual hardware needed to turn this into a
useful ARM64 virtual machine.

You can also check out the repository on my GitHub page and, if you have a
Raspberry Pi 4B, try the VMM yourself:

https://github.com/StjepanPoljak/kvm-rust