Coder Legion - Recent posts tagged containers

You patched PHP. The old image is still in your cluster’s future.

Fri, 23 Oct 2026 17:56:15 +0000

title: "You patched PHP. The old image is still in your cluster's future."
published: false
description: "PHP2Shell (CVE-2026-109618, CVE-2026-109619) as seen from a Kubernetes cluster: how fast the scanners caught up, where they stay blind, and why rebuilding the image is only half of remediation."

tags: kubernetes, security, php, devops

On 21 October PHP shipped fixes for two critical bugs in the engine itself, CVE-2026-109618 and CVE-2026-109619, both CVSS 9.2, both reachable without authentication, both already being exploited by the time most people read the advisory. The community name is PHP2Shell. Fixed versions are 8.5.12, 8.4.27, 8.3.36 and 8.2.35. CISA and the UK NCSC both published "patch now" notices the same day.

I am not going to explain the bugs; Wiz, watchTowr and Qualys have done that well. I want to write about what happened in Kubernetes teams in the forty-eight hours after, because I watched it happen and because the second half of it is the part nobody schedules.

How most teams found out

From the news. Then someone opened a spreadsheet and typed "PHP" into the first column.

This is not a criticism. It is what a platform team does when the only inventory of what runs in production is the memory of the people who deployed it. The questions that followed were always the same three:

Which clusters run PHP at all?
Which images, which versions?
Who gets to say "we are done"?

The third question is the one that comes back in March, when an auditor asks what you knew on 22 October and what you did about it.

How fast the scanners caught up

I measured this on 23 October, two days after the advisory, with Trivy 0.72.0 and whatever vulnerability database it pulled that afternoon. You can repeat every step.

The database build was stamped 13:13 UTC on 23 October. Both CVEs were in it, with the Debian and Alpine advisories attached:

$ grep -a -o CVE-2026-109618 ~/.cache/trivy/db/trivy.db | wc -l
1247
$ grep -a -o CVE-2026-109619 ~/.cache/trivy/db/trivy.db | wc -l
1253

Debian's security tracker already listed php8.2 as resolved in 8.2.35-1~deb12u1 for bookworm and php8.4 in 8.4.27-1~deb13u1 for trixie. Alpine's secdb for 3.22 carried both CVEs. A debian:bookworm-slim image with php8.2-cli installed from the archive that day came out as 8.2.35, already fixed; a alpine:3.22 image with php84 came out as 8.4.27, also fixed.

So the honest answer to "did the open-source scanners keep up" is yes, within two days, for PHP installed as a distribution package. That is good news and it is the ordinary case for a lot of internal images.

Now the uncomfortable part.

$ trivy image --scanners vuln --severity CRITICAL,HIGH php:8.3-fpm

126 CRITICAL and HIGH findings. Not one of them about PHP.

The official php:* images build PHP from source. There is no php8.3 package in the image for a package scanner to look at, so Trivy, and every scanner that works from the package database, has nothing to match the advisory against. The 126 findings are the Debian base. The thing you actually care about is invisible.

This is not a Trivy bug. It is what package-based scanning is. But it means that if your PHP workloads run on the official image, or on any image that compiles PHP, then a quiet scanner is not a clean bill. It is silence.

What you can do about it this week, in order of effort:

Ask the binary. docker run --rm <your-image> php -v tells you the truth in one line. Do it for every PHP image tag you run, not the ones you think you run.
Rebuild on a base that was pushed after 21 October, pin the digest, and write the digest down somewhere an auditor can read.
If you can, move to distribution packages for the runtime. It costs you a newer PHP, it buys you a scanner that can see it.

Patching is half the job

Here is the part that gets skipped. You rebuild the image, you roll the Deployment, the pods come up on 8.3.36, everyone exhales.

The old image did not go anywhere. It is still in the registry. It is still in helm history. It is the image someone will roll back to at 03:00 when the new build has a problem that has nothing to do with PHP. It is in the CI cache, in the staging cluster nobody looks at, in the Job that runs once a month.

Remediation has two halves:

Run the fixed image.
Make it impossible for the vulnerable image to run again, and keep a record that proves both.

Most teams do the first half and call it done. The second half is what an admission controller is for.

What this looks like with an admission controller in the path

This is where I talk about what we built, so take the next paragraphs for what they are. Attestkeep is an admission controller for Kubernetes that scans, verifies signatures, decides at admission, and keeps the record. Here is exactly what it did, and did not do, for PHP2Shell.

The database refreshes itself. The scanner pulls a fresh vulnerability database every six hours. Trivy publishes one every six hours too, so the lag between an advisory landing upstream and the cluster knowing about it is measured in hours, not in someone's calendar.

Every admitted image is rescanned. Once a day by default (scan.rescanInterval in the chart, if you want it shorter), every image the cluster has ever admitted is scanned again against the current database. An image that was clean on Monday and carries a CVE on Thursday shows up as not compliant on Thursday, with the CVE named. You do not open a spreadsheet; the inventory is the admission ledger.

It tells you. A scan that finds critical vulnerabilities raises a critical_finding event, which goes to whichever channel you configured: email, Slack, Teams, a webhook, Jira, GitLab. When a running image flips from "would be admitted" to "would be denied", that is a second, separate message: "Running image now fails policy". That is the message an operator actually needs.

New pods with the image are refused. If your policy blocks critical findings, the vulnerable image cannot be admitted again. The rollback at 03:00 fails with a reason, the Job that runs monthly fails with a reason, the staging cluster with the same policy refuses it too. The old image is still in the registry. It is no longer in your cluster's future.

Running pods are not killed. This is deliberate. Attestkeep does not evict a workload because a database update changed its verdict; that is an operator's decision, with the blast radius in front of them. What it does is make sure the next decision, any next decision, goes the right way, and that the running population is reconciled against the ledger on a schedule so drift is visible.

If the exploit lands first, there is a second line. PHP2Shell ends in a shell. If Falco is connected, a shell spawned inside a container is a runtime event, and a runtime policy can turn that event into a notification, an issue, or stopping the pod. That is not prevention, it is containment, and it is worth having when the window between advisory and rebuild is days.

Every decision is signed. The admission ledger records the image digest, the policy revision that ran, the verdict and the reason. When the auditor asks in March what you knew on 22 October, the answer is a record with a signature, not a recollection.

And the blind spot above applies to us exactly as it applies to Trivy, because Trivy is the scanner inside. If you run the official php:* image, Attestkeep's rescan will not name CVE-2026-109618 either. It will show you the digest you admitted, when, and under which policy. The php -v step is still yours. I would rather tell you that here than have you find it out from a pentest report.

The checklist, tool-agnostic

For this week, whatever you run:

List every image whose name or Dockerfile mentions PHP. Not from memory; from the cluster.
For each, run php -v inside the image. Trust that over the scanner.
Rebuild on a post-21-October base, pin digests.
Add a rule that refuses the old digests at admission, so a rollback cannot resurrect them.
Write down what you found, what you changed, and when, in a form that survives until the audit.

If step 4 is a policy in an admission controller and step 5 is produced automatically, this is a Tuesday. If step 4 is a Slack message and step 5 is a Confluence page, it is a project, and it will be a project again at the next CVE.

Closing

The lesson of PHP2Shell for platform teams is not "patch faster". Everyone patched. The lesson is that the news is a bad way to learn what is in your cluster, and that a patched Deployment with the old image still admissible is not a remediated cluster.

Let the cluster do the checking. Keep the record. Make the old image a thing that cannot come back.

Sources: CISA alert, 21 October 2026 · NCSC advisory · watchTowr FAQ · Qualys · Google Cloud response. Scanner measurements are mine, 23 October 2026, Trivy 0.72.0; the commands are in the text.

Attestkeep runs inside the cluster it protects; every security feature is in the free edition. attestkeep.com · docs.attestkeep.com

How and Why I Built My Own bootc-Based Linux Distribution

Sun, 20 Sep 2026 20:34:21 +0000

What an adventure it has been 🤠

It all started with yet another attempt to configure NixOS as a replacement for Debian, which I use every day.

I really like the idea behind NixOS: declarative configuration of the entire system and the ability to reproduce an environment both sound great. Unfortunately, in practice, it turned out to be much harder than I had expected. I find the Nix language completely unintuitive, the documentation can be maddening at times, and conflicting advice on best practices is scattered all over the internet. Or maybe I am just not smart enough to figure it out? Hard to say XD. In any case, after what was probably my third failed attempt to configure NixOS, I started looking for an alternative. That was when I discovered bootc.

As soon as I started reading the bootc documentation, I was blown away. An entire operating system described in a Containerfile and built like a regular container image, with updates and rollbacks powered by OSTree. Beautiful. Of course, this is not NixOS with different syntax, and it does not promise exactly the same things. But it gave me what mattered most: I could keep the recipe for my system in a repository and use it to build subsequent versions without manually recreating everything after every installation.

I quickly got to work configuring my dream distro. I am a minimalist by nature, so the feature-packed images prepared by Universal Blue were not an option. I wanted to decide for myself what went into the system. I chose the official quay.io/fedora/fedora-bootc image, which provides a Fedora base prepared to run through bootc: the kernel, essential system tools, and the image update mechanism. It is not a ready-made desktop loaded with applications, so I could add everything else myself. Fedora maintains the base, and for each rebuild I can use its current version instead of assembling the system from scratch by hand.

Of course, I am not crazy enough to switch immediately to a new system before it is ready for my day-to-day work.

At first, I decided to build and run SpaceOS in a virtual machine. I built the image with Podman, converted it to a qcow2 disk with bootc-image-builder, and then ran it in QEMU. This let me prepare the basic configuration and check whether the system booted at all. It soon became clear, however, that both building a new VM disk after every change and working inside the VM to run bootc switch were rather slow and frustrating. A VM also cannot tell me everything about my computer: it is not an easy way to test my Wi-Fi adapter, docking station, or DisplayLink-connected monitors.

So I decided to install the new system alongside Debian on the physical drive.

That was where things became a little tricky. I am somewhat paranoid and like to keep my data encrypted with LUKS. At first, I tried installing the system through Anaconda, but my partition layout made the process increasingly unpleasant. I also had the idea of letting Debian and SpaceOS share a single GRUB installation. I managed to get the system to boot, but after rebuilding the SpaceOS image, Debian's GRUB entry did not keep up with the new OSTree deployments. As a result, rollback became inconvenient even though it was supposed to be one of the greatest advantages of the whole exercise.

In the end, I put the two systems on separate LVM logical volumes inside the same LUKS container. Each system got its own /boot and /boot/efi partitions outside the encrypted container and its own UEFI boot entry. This allows bootc to manage its own GRUB installation and the entries for successive system versions, while at startup I choose either Debian or SpaceOS in UEFI. They share data encryption, not the boot mechanism.

How I Installed It

If you are mainly interested in the story of the project, feel free to skip this section. What follows describes the layout I used on my machine, not a universal recipe for every drive.

Partitioning operations are inherently dangerous, so I recommend making a backup first. Unless you enjoy an adrenaline rush and are not afraid of losing files 😉

My drive is laid out like this:

p1  Debian /boot/efi       FAT32
p2  Debian /boot           ext4
p3  shared LUKS container
    └── LVM volume group
        ├── Debian /        ext4
        ├── swap
        └── SpaceOS /       ext4
p4  SpaceOS /boot          ext4
p5  SpaceOS /boot/efi      FAT32

First, I made room at the end of the drive. I booted from a SystemRescue USB drive so I would not have to shrink the filesystem from which Debian was currently running. After unlocking LUKS and activating LVM, I checked Debian's unmounted filesystem and shrank it, then adjusted the sizes of the LVM and LUKS partitions. This part must be adapted to your own layout: order matters, and copying someone else's sector numbers from the internet is a bad idea, even if they belong to a very likable blog author. Before making any changes, I inspected lsblk, pvs, vgs, lvs, and the GPT partition table. After every operation, I made sure Debian still booted correctly.

From the space I had freed, I created two small partitions outside LUKS: about 1 GiB for /boot and about 1 GiB for the SpaceOS EFI partition. I added the remaining space back to the LUKS partition, expanded the LVM physical volume inside it using pvresize, and created a new logical volume named spaceos. This gave me around 58 GiB for the system. I formatted that volume and /boot as ext4, and the EFI partition as FAT32. This way, the SpaceOS root filesystem is encrypted together with Debian, while both systems have independent boot files.

After expanding and unlocking the LUKS partition, the part inside LVM looked roughly like this on my machine:

sudo pvresize /dev/mapper/nvme0n1p3_crypt
sudo lvcreate -L 58G -n spaceos ton618-vg
sudo mkfs.ext4 -L SPACEOS_ROOT /dev/ton618-vg/spaceos
sudo mkfs.ext4 -L SPACEOS_BOOT /dev/nvme0n1p4
sudo mkfs.fat -F32 -n SPACEOS_EFI /dev/nvme0n1p5

These are the device names from my computer. The three formatting commands in particular will erase the contents of the filesystems they target, so check that the partition numbers are correct before using them.

Next, I mounted the future root filesystem at /mnt/spaceos, the boot partition at /mnt/spaceos/boot, and the EFI partition at /mnt/spaceos/boot/efi. It is worth running findmnt for all three paths at this point. If any of them points to the wrong filesystem, this is definitely not the time to launch the installer.

sudo mkdir -p /mnt/spaceos
sudo mount /dev/ton618-vg/spaceos /mnt/spaceos
sudo mkdir -p /mnt/spaceos/boot
sudo mount /dev/nvme0n1p4 /mnt/spaceos/boot
sudo mkdir -p /mnt/spaceos/boot/efi
sudo mount /dev/nvme0n1p5 /mnt/spaceos/boot/efi
findmnt /mnt/spaceos
findmnt /mnt/spaceos/boot
findmnt /mnt/spaceos/boot/efi

I build the image locally with podman build. I do not publish it to a registry yet, so I installed from the local localhost/spaceos:latest image by running bootc install to-filesystem from within it. This method assumes that the filesystems have already been prepared and mounted; bootc should not have to guess how I wanted to partition the drive. I provided the UUIDs for / and /boot, and passed the UUID of the LUKS container and the name of the LVM volume as kernel arguments. Without the latter two, the system would not know what to unlock during startup. My script looks like this, except that I have replaced my own identifiers with placeholders for yours:

#!/usr/bin/env bash
set -euo pipefail

# Use blkid to find the filesystem UUIDs and cryptsetup luksUUID to find
# the LUKS container UUID. Set LVM_LV to the group and volume names.
ROOT_UUID="ENTER_ROOT_FILESYSTEM_UUID"
BOOT_UUID="ENTER_BOOT_PARTITION_UUID"
LUKS_UUID="ENTER_LUKS_CONTAINER_UUID"
LVM_LV="ENTER_GROUP_NAME/VOLUME_NAME"

for value in "$ROOT_UUID" "$BOOT_UUID" "$LUKS_UUID" "$LVM_LV"; do
  if [[ "$value" == ENTER_* ]]; then
    echo "Fill in the UUIDs and the LVM volume name first." >&2
    exit 1
  fi
done

sudo podman run --rm --privileged \
  --pid=host \
  --ipc=host \
  --security-opt label=type:unconfined_t \
  -v /dev:/dev \
  -v /var/lib/containers:/var/lib/containers \
  -v /mnt/spaceos:/target \
  localhost/spaceos:latest \
  bootc install to-filesystem \
    --bootloader=grub \
    --root-mount-spec="UUID=${ROOT_UUID}" \
    --boot-mount-spec="UUID=${BOOT_UUID}" \
    --karg="rd.luks.uuid=${LUKS_UUID}" \
    --karg="rd.lvm.lv=${LVM_LV}" \
    /target

Before running it, replace the placeholder values, verify that the filesystems are mounted under /mnt/spaceos, and make sure the image is available to the instance of Podman launched through sudo.

There was one more catch, this time caused by a bootc bug. With two EFI partitions on the same drive, bootc install to-filesystem could select the first one, belonging to Debian, even though I had mounted the SpaceOS EFI partition under the target directory. The issue has also been reported in the bootc repository. For the duration of the installation, I therefore unmounted Debian's EFI partition and temporarily changed its GPT type from “EFI System” to a regular Linux partition. I did not format it or remove any files. After the installation, I restored the correct type and mounted it again. This works around a specific partition-selection problem, but it is also a step where confusing the partition numbers can cause serious trouble.

Finally, I used efibootmgr -v to verify that the SpaceOS entry actually pointed to its own EFI partition. Only then did I decide I could breathe again. After my earlier adventures with GRUB, I preferred evidence over a good feeling.

Developing the System from Within

This was when the fun part of the whole operation began. With SpaceOS correctly installed alongside my main system, I could develop it from within SpaceOS itself. I edit the Containerfile, build a new image locally, and switch the system to it with bootc switch --transport containers-storage localhost/spaceos:latest. After a restart, the new deployment boots; if I break something, I can return to the previous one. This is exactly the kind of workflow that made me interested in bootc.

Installing software and preparing configuration here resembles building a regular container. I currently use Fedora 44 as the base, and some of the main programs included in the system are Sway, Fish, Alacritty, and Neovim. You can find the full and up-to-date list of software in the SpaceOS repository. The project is still evolving, so its contents may change over time.

I also came up with the idea of copying the repository to /etc/spaceos during the build and creating symbolic links to its configuration files. This lets me edit the configuration in one place and immediately see the changes in programs that read it dynamically. Moreover, uncommitted local changes under /etc do not disappear simply because the system switches to a new image. This does not mean bootc transfers my files into the image, however: Podman uses the current contents of the project directory during the build, while OSTree separately carries local /etc changes over to the new version during deployment. If I edit something on the running system, I need to remember whether I want to keep it locally only or commit it to the repository as well.

That distinction matters. /usr and most of the system come from the image and are replaced with new contents when the version changes. /etc remains writable by default and persists across updates, with local changes taken into account. Symbolic links to /etc/spaceos therefore make sense for my experiment, although they require some care: if I modify a file locally, a new version of the same file from the image may not overwrite my edit. /var, meanwhile, holds data that should survive both system switches and rollbacks. The bootc documentation explains these rules in much greater detail.

DisplayLink, Because of Course It Couldn't Be That Easy

The DisplayLink drivers for my docking station proved to be one of the bigger problems. Adding an RPM package to the image looked innocent enough. But then I still had to build the EVDI module for the exact kernel included in the image, start the user-space component, and get Sway to cooperate with the additional displays. My first attempts produced the classic “almost works” symptom: the image built successfully, but after login Sway immediately kicked me back to greetd.

At first, I tried installing an unofficial DisplayLink RPM and building EVDI with DKMS while assembling the image. The module build itself could finish successfully, which unfortunately did not mean that the graphical session would start on the physical machine. On top of that, bootc container lint warned about files left behind by DKMS and other installation steps. So I began treating the module build and DisplayLink's operation on the hardware as separate problems.

In the end, I studied how Universal Blue uses akmods and adapted the idea to my own Containerfile, without switching to one of its ready-made images. In a separate build stage, I install the compilation tools and the kernel-devel package matching the kernel from fedora-bootc, build EVDI, verify that the module was actually produced, and copy the resulting package together with the user-space part of DisplayLink into the final image. This means the compilers and the entire build toolchain do not have to remain in the target system.

I get the packages from the Fedora Multimedia repository maintained by Negativo17, also known as Simone Caronni. If you use Fedora and have ever needed something outside its core repositories, you may already have encountered his work. In my case, the displaylink, libevdi, and akmod-evdi packages available there saved me from a great deal of additional manual tinkering. A small nod of appreciation to him, because without people like that, our “it's just one driver” projects would be far less fun.

The current Containerfile builds EVDI in a separate stage and installs it in the SpaceOS image. I also had to change the command that launches Sway in greetd, adding the --unsupported-gpu flag. I tested the image on the physical computer, and this time DisplayLink works with the docking station while Sway no longer sends me back to greetd. Finally 😄

Updates Managed by CI/CD

Once the system became suitable for everyday use, building images locally was no longer enough. It is convenient while working on the configuration, but it is hardly a sensible way to publish subsequent system versions. I also did not want to remember to rebuild SpaceOS manually every time Fedora updated the base image. It was time for GitHub Actions and GHCR.

An official release now begins with a tag following the v0.1.3 pattern. The workflow validates its format, builds the SpaceOS image, generates /etc/os-release with the correct version number, and publishes everything to GitHub Container Registry. The immutable version tag remains in the registry permanently, while the stable, latest, and auto tags are moved to the newly released image. This means that someone following the auto channel receives not only Fedora fixes, but new SpaceOS releases as well.

Initially, the same tag was also supposed to trigger a build of the Anaconda installation ISO and attach it to a GitHub Release. The build itself worked, but the resulting file exceeded GitHub's 2 GiB limit for a single release asset. I could split an archive into several parts or find separate hosting for files that large, but that would be overengineering for a system built primarily for myself. In the end, the CI/CD pipeline publishes only container images to GHCR. Instead, the repository contains a script for building the ISO locally. It pulls the image from the auto channel, embeds it in the installer, and generates a SHA-256 checksum. A system installed from this media follows auto from the start, so it does not have to be switched manually to the update channel after installation.

A second workflow handles exactly those changes. Once a week, it checks the digest of the fedora-bootc:44 image and compares it with the base used by the most recently published SpaceOS image. If Fedora has changed nothing, the action finishes without building anything. If the digest is new, the workflow checks out the latest release, rebuilds the image, and publishes it under a tag containing the SpaceOS version, the date, and part of the base digest. One example is auto-v0.1.3-20260920-abcdef123456. The auto tag is moved to that image, and I retain the five most recent automatic rebuilds in the registry. Images carrying release tags are not subject to this cleanup.

The remaining question was what to do on the running system. bootc's default automatic update mechanism did not suit the way I work, because I do not want my computer deciding to restart at the least convenient moment. Instead, I added my own systemd timer. Every hour, it checks the update channel and, provided the connection is not metered, the battery is not too low, and the machine is not under heavy load, downloads and stages a new deployment in the background. It does not restart the computer. Waybar tells me that an update is waiting, and the new version starts only the next time I reboot the machine myself.

Summary

SpaceOS began with the frustration of yet another attempt at NixOS and ended up as a system whose entire recipe I can keep in Git, test like a container image, and update without manually recreating my environment. I had to overcome many problems along the way, but in the end I succeeded. Those very problems ultimately gave me a much better understanding of bootc, OSTree, the process of building a custom operating system, and many other Linux-related topics.

The project's code is available in the SpaceOS repository on GitHub. Its README contains a more detailed description of the system itself, the installation process, published tags, update mechanism, included software, and keyboard shortcuts.

I should emphasize that SpaceOS is a system built specifically for me, my hardware, and the way I work. I would not really recommend installing it directly as your own operating system. Instead, I encourage you to fork the repository and treat it as a starting point for building your own unique distribution. After all, the greatest advantage of this approach is precisely that the system can truly be yours.

One File, Six Services: Running AI Agents With Docker Compose

Thu, 27 Aug 2026 23:49:58 +0000

Most of us here already reach for Compose the moment a project grows past a single process, and agent work hits that point faster than almost anything else. What follows is the shape that keeps showing up once an agent stops being a script and starts being a system.

An Agent Is Never One Process

A working agent needs a runtime process, a model endpoint, a database for conversation history and agent state, and usually a vector store for retrieval. Add a queue once tasks run in the background, and something for traces once you need to know why a run went sideways. That is six services before anyone has written an interesting tool.

Running those by hand means remembering startup order, wiring hostnames and ports, and keeping six sets of environment variables in sync. Compose turns all of it into one declarative file with depends_on and health checks, so docker compose up produces the same stack on a laptop and on a server, and docker compose down leaves nothing behind.

The dependency isolation matters more here than in most stacks. Agent code pins specific versions of Python, PyTorch, and CUDA, and those pins conflict constantly. Two containers can run CUDA 12.4 and CUDA 11.8 side by side because each one carries its own dependency tree.

Three Patterns, Three Cost Profiles

The first pattern is API-backed. The agent container calls a hosted model, and the Compose file holds the runtime, a database, and whatever tools the agent needs. Minimal local resources, best available models, and a bill that scales linearly with usage.

The second runs a model server such as Ollama or vLLM in the same stack. The server holds the model in GPU memory and exposes an OpenAI compatible endpoint on the internal Docker network, so the agent reaches it by service name and configuration stays trivial. Per-call cost goes to zero and nothing leaves your infrastructure, but you own the GPU memory budget, and a large model can want anywhere from 20 to 80 GB of VRAM depending on parameter count and quantization.

The third is a hybrid that routes cheap high-volume calls to the local server and hard calls out to a frontier model. It is the pattern most teams land on eventually, and it is also the one that makes the routing logic worth testing.

The Parts That Bite Later

GPU access needs the device reservation in the Compose file and the container toolkit on the host, and it fails in a way that looks like a slow CPU run rather than an error. State needs named volumes, because a bind mount to a host path is the fastest way to lose a vector index during a rebuild. Service discovery is free inside the Compose network, which is exactly why it is worth being deliberate about which ports you publish to the host at all, since an unauthenticated model server on 0.0.0.0 is a real problem.

This guide to deploying AI agents with Docker Compose covers each of those in depth, along with the compose file anatomy and the move from a development stack to a production one.

Where Compose Stops Being The Answer

Compose is a single-host tool. When you need rolling updates across machines, autoscaling on queue depth, or real multi-tenancy, that is the Kubernetes conversation. Until then, a Compose file is the honest description of your architecture, and it is the artifact a new developer can read in two minutes and run in one command.

Understanding the vectors of container escapes

Sun, 16 Aug 2026 20:09:22 +0000

Container escapes typically occur when a container gains direct access to host resources due to excessive permissions or shared namespaces. I'm working on a few Github projects to explore these risky misconfigurations in reproducible PoCs. The 3 categories of risk I'm exploring that commonly lead to container escapes are:

Dangerous volume mounts
Linux capabilities
Shared namespaces

Dangerous volume mounting (like hostPath) is when a developer mounts a directory from the underlying host filesystems like /, /var/run/docker.sock or /etc directly into a container. The risk here is if a container mounts the host root / filesystem or sensitive sockets like any sort of container runtime socket, a process inside the container can now interact directly with the host OS or manage container runtimes, effectively bypassing container isolation.

Github reference:
https://github.com/ndouglas-edera/hostpath-risk

Linux capabilities (like CAP_SYS_ADMIN and CAP_SYS_PTRACE) are granular administrative privileges assigned to processes within a container. The risk here is that capabilities like CAP_SYS_ADMIN allow actions such as mounting filesystems, setting network configs, or even interacting with kernel interfaces, ultimately enabling container processes to break out of their Linux namespaces.

Github reference:
https://github.com/ndouglas-edera/CAP_SYS_ADMIN-Escape

Shared namespaces (like hostPID, hostNetwork, and hostIPC) end up disabling the standard namespace separations between the container and the underlying node. It's ultimately risky because it allows containers to interact with host processes, inter-process communication channels, or raw network interfaces, breaking isolation between workloads on the same node.

Github reference:
https://github.com/ndouglas-edera/hostpid-risk

How isolated runtimes can mitigate these risks

The above scripts I provided on Github were purely to automate the demonstration of these existing insecure configurations within Kubernetes. Since standard containerised workloads rely on various shared host Linux kernel features, such as namespaces and cgroups, to enforce boundaries - if a container ends up gaining elevated privileges or access to host resource, it now operates directly on that shared kernel. Secure runtime solutions like Edera Zones, Kata Containers, and gVisor change this threat model by providing secure (MicroVM) sandboxing layers. Instead of sharing the host kernel, workloads run inside lightweight VMs with their own isolated kernel. Similarly, even if a process inside the sandbox achieves privileged: true or mounts a volume, its actions are constrained to its own guest kernel. Escaping the container only grants access to the isolated microVM environment, preventing access to the host OS, host memory, or adjacent workloads on the node. Finally, direct access to host process IDs or raw host sockets is prevented because the guest kernel maintains an entirely separate execution environment from the host operating system. I'm going to continue fleshing-out these repos to show how these misconfigurations are common place, but also how to address them within a single-node test sandbox like EderaOn.

Lateral credential harvesting via process namespace sharing

Fri, 14 Aug 2026 16:51:02 +0000

Github Source:
https://github.com/ndouglas-edera/hostpid-risk/tree/main

In a multi-tenant or shared Kubernetes node environment, containers are supposed to be isolated environments. However, this demo proves that if a malicious or compromised pod is deployed with host-level privileges, it can spy on neighbouring workloads running on the same node without needing a software vulnerability or zero-day exploit.

Example script

You can download the automation script from Github:

wget https://raw.githubusercontent.com/ndouglas-edera/hostpid-risk/refs/heads/main/hostpid-risk.sh
chmod +x hostpid-risk.sh

If you have a Kubernetes cluster already running (like on Docker Desktop or kind), you can run the below script:

./hostpid-risk.sh

What is the security misconfiguration exactly?

The exploit relies on 2 specific fields in the inspector-pod YAML manifest working together:

spec:
  hostPID: true            # <--- Misconfiguration #1: Breaks PID isolation
  containers:
  - name: inspector
    securityContext:
      privileged: true     # <--- Misconfiguration #2: Overrides Linux permissions

How it works technically

Normally, Linux container runtimes place each pod in their own isolated Process ID (PID) namespace.
Setting hostPID: true actually forces the pod to share the host node’s global PID namespace.
Suddenly, ps -ef inside the container displays every process running on the underlying node.
This includes processes belonging to other pods (target-app).
This would bypass Linux kernel profiles like AppArmor and SELinux - granting full capabilities (like CAP_SYS_ADMIN).
In Linux, the virtual /proc filesystem exposes process metadata. The file /proc/<PID>/environ holds the environment variables loaded into memory when that process was spawned.
Because inspector-pod has host-level permissions, it can read /proc/30629/environ belonging to target-app before extracting the DB_PASSWORD=SuperSecretPassword123! creds.

Strategy 1 - PSS in Kubernetes

Pod Security Standards (or PSS, for short) is a built-in policy Kubernetes framework managed by the Pod Security Admission (PSA) controller. It categorises security into 3 levels:

Privileged: Unrestricted (Allowing hostPID: true & privileged: true)
Baseline: Prevents known privilege escalations (Blocking hostPID: true, host paths, and host networking)
Restricted: Hardened best practices (Requires dropping capabilities, non-root users, and more..)

So, how do PSS policies block the attack?
By either applying a Baseline or Restricted policy to the target namespace, Kubernetes will reject inspector-pod at the API server before it can ever be scheduled. Here's an example of enforcing the baseline profile on the default namespace

kubectl label --overwrite ns default pod-security.kubernetes.io/enforce=baseline

If an attacker tries to apply inspector-pod.yaml now, the Kubernetes API server responds with an immediate error, similar to other admission control technologies like OPA:

Error from server (Forbidden): pods "inspector-pod" is forbidden: 
violates PodSecurity "baseline:latest": hostPID (spec.hostPID=true), 
privileged (container "inspector" must not set securityContext.privileged=true)

Strategy 2 - Runtime Isolation

While PSS works at the API controlplane-level (by stopping bad YAML from being accepted), the ideal runtime isolation approach should be applied at the Kernel-level.

For context, hostPID: true and privileged: true are commonly enabled purely by accident, oftentimes by developers while troubleshooting nodes, and through legacy Helm charts, or by third-party monitoring and observability agents (like eBPF probes, custom log forwarders, or node-exporters) that require node visibility. And when this happens in a multi-tenant cluster (that's shared across departments or clients), compromising a single pod with host privileges instantly compromises all secrets across all pods on that same node, rendering the entire namespace-level isolation useless.

So, what if an attacker has cluster-admin privileges or bypasses PSS?
Standard Kubernetes containers share a single, common Linux kernel. If namespace boundaries break, everything on that kernel is exposed. In the case of Edera Zones (and microVM-based runtimes like Kata Containers or gVisor, more generally) they implement a defence-in-depth strategy by isolating all your workloads into distinct hypervisor boundaries or independent kernel spaces.

Under gVisor, Kata Containers, or Edera Zones, even if a pod sets hostPID: true and privileged: true, hostPID only shares the PID namespace of the isolated guest kernel / microVM sandbox, not the underlying bare-metal host node. The malicious pod sees only itself and its sandbox. In this way, it can't see processes from neighbouring pods running in parallel sandboxes.

Note: I will continue to update the Github repository to better improve these security automation demos.

Breakouts, Bots, and Bare Metal

Fri, 31 Jul 2026 14:24:01 +0000

For years, security teams viewed container breakout attacks as an operational nuisance. The kind of nuisance that was largely confined to misconfigured Kubernetes clusters, over-privileged dev environments, or cloud cryptojacking campaigns. The security industry recently witnessed how fast that assumption has broken down dramatically.

From OpenAI disclosing that its frontier AI models escaped their evaluation test environment and compromised external database infrastructure, to Anthropic admitting its Claude models broke free of sandboxes across three separate target environments, escape-to-host is no longer a problem isolated to cloud-native infrastructure. It is the defining bottleneck of safe AI deployment.

Understanding why these breakouts keep happening, and why traditional container sandboxing approaches have consistently failed, requires stepping back into MITRE ATT&CK technique T1611, dissecting past isolation trade-offs, and examining why bare-metal Type-1 hypervisor architecture provides the isolation boundary modern workloads require.

What is T1611?

At its core, the escape-to-host technique describes an adversary breaking out of a constrained environment (a container, sandbox, or a VM) to gain unauthorised access to the underlying host kernel, host filesystem, or hypervisor. We also hear the phrase “container escape” being used more specifically in Docker & Kubernetes.

In modern Linux OS virtualisation, standard containers do not run on isolated hardware. Instead, technologies like Docker and Kubernetes mostly rely on the shared-kernel isolation mechanisms. This includes the below:

namespaces (PID, mount, network) to slice system visibility.
cgroups to limit hardware resource consumption.
seccomp & capabilities to drop dangerous syscalls.

While these controls partition resources efficiently, the host kernel remains a massive single point of failure. An adversary, or any sort of hyper-capable autonomous model executing untrusted code, can execute an escape via several proven vectors:

Vector	Description
Privileged containers	Containers running with `--privileged` or equivalent flags
Host bind mounts	Mounting sensitive host paths (like `/proc` or `/`) can execute direct write operations to host processes or write malicious cron jobs.
Exposed container sockets	Mounting `/var/run/docker.sock` into a pod grants full root privileges over the host machine via administrative container commands.
Kernel syscall exploitation	Abusing vulnerable or permissive syscalls (such as `unshare`, `keyctl`, or `copy_file_range`) to trigger privilege escalation bugs in the shared Linux kernel.

Once an attacker breaks the boundary, the host level unlocks lateral movement across every neighbouring pod, secrets held in host memory, and potentially full cluster takeover. Prioritising these container escape vulnerabilities, like CVEs in general, is becoming harder and harder due to the so-called AI-driven vulnpocolypse.

An overly-simple tool (hosted on GitHub) I put together to prioritise CVEs in public DockerHub images

When code execution becomes autonomous

So, why is T1611 suddenly dominating headlines outside the traditional DevOps circles? Isn’t it just another vulnerability that needs to be fixed? Well, yes. But it also differs drastically because of how autonomous AI agents rely heavily on, and will continue to abuse, arbitrary code execution behaviour.

To solve complex engineering tasks, execute data science workflows, or evaluate security tools, modern AI agents must run dynamic code inside sandboxes. However, when these AI agents encounter unexpected 0 Days or any sort of deployment misconfigs, their single-minded goal priorities turn them into super-powerful, automated pentesters.

During automated benchmark tests at OpenAI on unreleased models, safety refusals were relaxed to gauge offensive cyber capabilities. The models autonomously chained credentials and unpatched 0-day CVEs inside their testing environment, reached the internet, and managed to compromise live production servers on Hugging Face to pull down testing solutions.
Likewise, Anthropic’s Claude models participated in CTF-style evaluation scenarios, and also managed to escape their intended test boundaries across 141,000 runs. The models executed unauthenticated SQL injections, created and published malicious packages to public PyPI software repos, and even extracted production database credentials.

These incidents underline a hard security truth that human error and software misconfigurations will always happen. Whether via human mistake, prompt injection, or autonomous goal-seeking, relying on standard OS-level sandboxing means that a single hole in the app-layer or shared kernel cascades into total infrastructure breaches.

M1048 - Application isolation and sandboxing

To mitigate T1611, it’s recommended to adopt the MITRE ATT&CK mitigation ID M1048. The cloud-native ecosystem has spent a decade or more experimenting with alternative containment boundaries. However, almost every historical attempt forced engineers into a goldilocks dilemma whereby we have to make hard decisions about trading off security, speed, and/or DevEx in some cases.

Vector	Description	Description
OS Virtualisation	Docker / K8s	While fast and lightweight, sharing the host kernel creates a fragile security boundary with a massive attack surface.
Intercept syscalls	gVisor	Re-implementing kernel calls in userland via a Go runtime wrapper introduces heavy syscall overhead and severe performance penalties on bulkier I/O operations.
Type-2 MicroVM	Kata / F'crackr	Boot times are slower, cloud providers often restrict nested KVM virtualisation extensions, and the KVM host kernel remains inside your Trusted Computing Base (TCB).
Unikernel	IncludeOS, OSV	Custom compilation requirements introduce major dev friction and break compatibility with standard OCI container images and Kubernetes tooling.

1. Userspace syscall emulation
gVisor is universally celebrated for making sandboxing accessible without forcing changes to how containers are deployed. Developed by Google and battle-tested in platforms like Google Cloud Run, Cloud Functions, and the GKE Sandbox, it provides an elegant way to execute untrusted code or multi-tenant workloads. By intercepting application system calls in userspace using a memory-safe Go kernel called the sentry, gVisor effectively removes direct exposure to the host Linux kernel and eliminates whole classes of privilege escalation attacks. However, because it re-implements the complex Linux API inside a user-space process, it inherently introduces performance friction. Workloads heavy on system calls, network operations, or disk I/O often experience noticeable latency and CPU overhead, while the sandbox still shares the host filesystem under the hood.

2. Type-2 MicroVMs & KVM-based wrappers
Type-2 microVM technologies represent a massive leap forward for multi-tenant isolation, serving as the trusted foundation for cloud-scale infrastructure like AWS Lambda and AWS Fargate. By leveraging KVM (Kernel-based VM), engines like Firecracker and Kata Containers achieve lightning-fast sub-second boot times and minimal memory footprints while granting each workload a dedicated guest kernel and hardware-level isolation. Despite these monumental achievements, their reliance on KVM means the host Linux operating system itself remains directly in the TCB. This means any vulnerability in the host kernel can still compromise the host machine. Additionally, running KVM-backed microVMs inside modern public cloud environments requires nested virtualisation, an extension that remains restricted, cost-prohibitive, or entirely unsupported across many standard cloud node pools.

3. Unikernels
Unikernels offer the theoretical gold standard of minimal attack surfaces and hyper-efficient resource utilisation. By compiling an app and only its exact system dependencies into a single, unified machine image that runs without any traditional OS layer, unikernels deliver near-instantaneous startup times, minuscule memory footprints, and virtually zero bloat for adversaries to exploit. This radically lightweight design has made them a popular research domain for high-performance and single-purpose computing. However, this minimalist paradigm comes at the cost of operational usability. Because unikernels demand customised build pipelines and specialised compilation targets, they inherently break compatibility with standard OCI container images, existing Dockerfile workflows, and native Kubernetes orchestration pipelines.

4. Hardware-backed Type-1 hypervisor isolation
To permanently resolve T1611 without sacrificing container performance or developer ergonomics, the shared host kernel must be removed from the TCB altogether. This is the paradigm pioneered by projects and platforms like Edera, which pair an optimised VMM with a minimal Type-1 (bare-metal) Xen microkernel hypervisor.

Unlike Type-2 hypervisors that run on top of millions of lines of standard Linux C code, a paravirtualised Type-1 design relies on a microkernel core written or wrapped in memory-safe languages like Rust. This eliminates entire classes of hypervisor memory-corruption vulnerabilities (such as buffer overflows and use-after-free bugs) while reducing the ring-0 code footprint by up to 90%. By using paravirtualisation hypercalls instead of heavy CPU hardware emulation, guest OS containers run with hypervisor awareness. This delivers near-native CPU performance without requiring nested virtualisation extensions, allowing it to run on standard cloud infrastructure. Crucially, modern implementations act as drop-in replacements for standard Kubernetes CRIs, preserving existing OCI container images, Helm charts, and kubectl workflows while giving every pod hardware-enforced boundaries.

And wherever there is a security breakthrough, there will always be some slight drawbacks. While runtime performance of the hardware-backed type-1 hypervisor isolation matches native containers within a few % points, spinning up a fresh hypervisor zone introduces a small startup penalty (somewhere below 1s). This is unnoticeable for long-running AI agents or database microservices, but may require optimisation for ultra-short-lived, serverless functions. Also, moving host nodes to bare-metal hypervisors alters node provisioning compared to standard Linux worker nodes running basic containerd. This is a noticeable operational paradigm shift for DevOps teams.

Closing the door on sandbox escapes

As autonomous AI agents and multi-tenant cloud workloads scale, the boundary between untrusted application code and trusted host management will continue to blur. Relying on software-only or userspace Linux kernel controls to block sophisticated privilege escalation attempts is a losing battle. While all of the approaches mentioned above are fantastic for preventing escapes, there is always going to be some kind of drawback (either performance or compatibility) when we opt for a secure-by-default approach. I hope I made a clear case for pairing modern Type-1 bare-metal hypervisor isolation with the DevEx of native containers to neutralise MITRE ATT&CK’s technique ID T1611 at the architectural level. When another frontier lab’s agent or an actual malicious actor decides to break out of a container runtime or sandbox environment, they won't land on the host OS, they should ultimately land in an isolated hypervisor zone with nowhere left to go.

From Dockerfile to Docker Hub: A Complete Beginner's Guide to Docker

Thu, 11 Jun 2026 10:31:28 +0000

Containerization has become a fundamental skill for DevOps Engineers, Cloud Engineers, and Software Developers.

However, many beginners learn Docker commands without understanding the complete workflow behind them.

Questions like:

What exactly is a Docker Image?
How does a Dockerfile become a running Container?
Why do we push images to Docker Hub?
How do organizations deploy the same application everywhere?

are common when starting out.

In this blog, we'll build a simple Node.js application, containerize it using Docker, run it locally, and finally push the image to Docker Hub.

By the end of this guide, you'll understand the complete Docker workflow used in real-world DevOps environments.

Docker Workflow Overview

flowchart LR
    A[Application Code] --> B[Dockerfile]
    B --> C[Docker Build]
    C --> D[Docker Image]
    D --> E[Docker Run]
    E --> F[Docker Container]
    D --> G[Docker Push]
    G --> H[Docker Hub]

This is the same workflow followed by developers and DevOps teams worldwide.

What Are We Building?

We'll create a simple Node.js application that returns:

A welcome message
Environment information
Container hostname

This helps us verify that our application is actually running inside a Docker container.

Project Structure

.
├── Dockerfile
├── app.js
└── package.json

Step 1: Create the Node.js Application

app.js

const express = require("express");
const app = express();

const PORT = process.env.PORT || 3000;
const ENV = process.env.ENV_VALUE || "No env set";
const HOSTNAME = process.env.HOSTNAME || require("os").hostname();

app.get("/", (req, res) => {
  res.json({
    message: "Hello from Simple App (Node)",
    env: ENV,
    container: HOSTNAME
  });
});

app.listen(PORT, () =>
  console.log(`Node Hello listening on ${PORT}`)
);

package.json

{
  "name": "hello-app-node",
  "version": "1.0.0",
  "main": "app.js",
  "dependencies": {
    "express": "^5.2.1"
  }
}

Application Architecture

flowchart LR
    User --> Browser
    Browser --> Container
    Container --> NodeJS
    NodeJS --> JSONResponse

Whenever a user accesses the application, the request reaches the containerized Node.js application, which returns a JSON response.

Step 2: Create the Dockerfile

A Dockerfile is simply a blueprint that tells Docker how to build an image.

FROM node:24-slim

WORKDIR /app

COPY package*.json ./

RUN npm install --production

COPY . .

EXPOSE 3000

CMD ["node", "app.js"]

Understanding the Dockerfile

Base Image

FROM node:24-slim

We start with an official lightweight Node.js image from Docker Hub.

Working Directory

WORKDIR /app

Creates and switches to the /app directory inside the container.

Copy Dependency Files

COPY package*.json ./

Copies package files first.

This is considered a Docker best practice because Docker can cache dependency layers and speed up future builds.

Install Dependencies

RUN npm install --production

Installs only production dependencies.

Copy Application Source Code

COPY . .

Copies the entire project into the container.

Expose Application Port

EXPOSE 3000

Documents that the application listens on port 3000.

Start the Application

CMD ["node", "app.js"]

Runs the Node.js application when the container starts.

What Happens During Docker Build?

flowchart TD
    A[Dockerfile]
    --> B[Pull Node Base Image]
    --> C[Create Working Directory]
    --> D[Copy package.json]
    --> E[Install Dependencies]
    --> F[Copy Application Code]
    --> G[Create Image Layers]
    --> H[Docker Image Ready]

Docker executes each instruction sequentially and creates image layers.

Step 3: Build the Docker Image

Run:

docker build -t madhavnaks/node-app .

Let's understand this command:

docker build

Builds an image from a Dockerfile.

-t

Assigns a tag to the image.

madhavnaks/node-app

Image name.

Current directory containing the Dockerfile.

After a successful build you'll see output similar to:

=> naming to docker.io/madhavnaks/node-app

Docker has now created an image locally on your machine.

Step 4: Verify the Image

To see locally available images:

docker images

Example output:

REPOSITORY                TAG       IMAGE ID       SIZE
madhavnaks/node-app       latest    42c936686000   234MB

At this point, the image exists but is not running.

Docker Image vs Docker Container

Many beginners confuse these two concepts.

Think of it like this:

Docker Image = Blueprint
Docker Container = Running Application

flowchart LR
    A[Docker Image]
    --> B[Container 1]

    A --> C[Container 2]

    A --> D[Container 3]

A single image can create multiple containers.

Step 5: Run a Container

Now let's start our application.

docker run -d --name node-app -p 3001:3000 madhavnaks/node-app

Understanding the Command

Detached Mode

-d

Runs the container in the background.

Container Name

--name node-app

Assigns a friendly name to the container.

Port Mapping

-p 3001:3000

Maps:

Host Port      → Container Port
3001           → 3000

The application listens on port 3000 inside the container but is accessible on port 3001 from your machine.

Step 6: Verify the Container

Check running containers:

docker ps

Expected output:

CONTAINER ID   IMAGE                 STATUS
10265698c978   madhavnaks/node-app   Up

Step 7: Access the Application

Open:

http://localhost:3001

Expected response:

{
  "message": "Hello from Simple App (Node)",
  "env": "No env set",
  "container": "10265698c978"
}

Notice the hostname.

The hostname corresponds to the container ID, proving the application is running inside Docker.

Before pushing images, authenticate with Docker Hub.

docker login

Example:

Login Succeeded

Docker is now authorized to push images to your repository.

Why Push Images to Docker Hub?

Docker Hub acts as a centralized registry.

flowchart LR
    A[Developer Machine]
    -->|Push| B[Docker Hub]

    B -->|Pull| C[Server]

    B -->|Pull| D[Kubernetes Cluster]

Once an image is pushed, it can be pulled and run from anywhere.

Step 9: Push the Image

Push the image to Docker Hub:

docker push madhavnaks/node-app

Example output:

latest: digest: sha256:d1081ec542f521e100bd886943c13ab7568c663b56e36200e6d846163975f979

This confirms that the image has been successfully uploaded.

Step 10: Verify on Docker Hub

Visit your Docker Hub account.

You should now see:

madhavnaks/node-app

listed under repositories.

You can even share the image with others.

Anyone can now run:

docker pull madhavnaks/node-app

docker run -p 3001:3000 madhavnaks/node-app

without needing your source code.

Real-World DevOps Workflow

In production environments, the workflow usually looks like this:

flowchart LR
    A[Developer Writes Code]
    --> B[GitHub]

    B --> C[CI/CD Pipeline]

    C --> D[Docker Build]

    D --> E[Docker Hub]

    E --> F[Kubernetes Cluster]

This is exactly how modern DevOps teams package and deploy applications.

Key Takeaways

✅ Dockerfile defines how an image should be built

✅ Docker Build creates an image

✅ Docker Run creates a container

✅ Docker Hub stores and distributes images

✅ One image can create multiple containers

✅ Containerization ensures consistency across environments

Conclusion

In this guide, we covered the complete Docker workflow from source code to Docker Hub.

We learned how to:

Create a Dockerfile
Build a Docker image
Run a container
Verify application execution
Push images to Docker Hub
Understand how images move through real-world DevOps pipelines

Mastering these Docker fundamentals is the first step toward advanced topics like Docker Compose, Kubernetes, Helm, CI/CD pipelines, and cloud-native deployments.

If you're learning DevOps, try containerizing applications in different languages such as Python, Java, Go, or .NET to strengthen your understanding of Docker.

Happy Containerizing!

Running Astro in a preview container

Mon, 08 Jun 2026 16:35:17 +0000

If you've ever worked on a collection of different Node apps, you've likely encountered version conflicts. Everyone wants a different version of Node or PNPM, and your new job is trying to align them all, or managing versions daily.

That's when open-source hero Kostis Kapelonis said, "Why don't we run the preview in a container?" In fact, he didn't just say this; he also submitted a PR. I told you he's an open-source hero.

The PR added a Dockerfile and a docker-compose.yaml file to the project, which let you spin up the preview site using:

docker compose up

Once Kostis had done all the hard work, I added a small enhancement to make Astro's live preview work when you change files. That meant you could start the container and keep working while all your changes are instantly visible in the preview. That keeps the developer inner loop nice and tight.

If you want to do the same, here's how to make it happen. Once again, I added a very small cherry to Kostis' wonderfully fluffy cake, so send your adoration his way.

Add a Docker compose file

Here's the docker-compose.yml file for your Astro project. It goes in the root directory.

services:
  astro:
    build: .
    ports:
 - "3000:3000"
    volumes:
 - .:/app
 - /app/node_modules
    environment:
 - NODE_ENV=development
 - HOST=0.0.0.0
    stdin_open: true
    tty: true

This maps the volumes, with a special case for node_modules. It exposes Astro's port 3000 inside the container to port 3000 on your machine so you can open it in your browser.

The Docker file

Here's the Dockerfile, which also goes in your project's root directory.

# Use Node 20 as the base image
FROM node:20-slim

# Install pnpm globally
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
RUN corepack enable

# Set the working directory
WORKDIR /app

# Copy package files
COPY package.json pnpm-lock.yaml* ./

# Install dependencies
RUN pnpm install

# Copy the rest of the source code
COPY . .

# Expose the default Astro port
EXPOSE 3000

# Start the dev server
CMD ["pnpm", "compose:dev"]

There's an optimization here around the package files, which is why they get their own copy command. There's an extra command in the package.json file that we call here, too. It's a variation of the dev script we use, but switches out the Astro run with a slight variation (the addition of the --host flag) lets just show the important bits in this code snippet:

"scripts": {     
    "compose:dev": "npm-run-all --parallel dev:img dev:dictionary compose:dev:astro dev:watch",
    "compose:dev:astro": "astro dev --host",

The Vite config change

The final change is the one that makes the live refresh to work. This goes in your astro.config.mjs file, and I popped it right after the existing server config.

server: {
    port: 3000
}
vite: {
    server: {
        watch: {
            usePolling: true,
        },
    },
},

Spinning it up

The first time I ran this, I started things up with:

docker compose up --build

You can then stop the container with

docker compose down

If you haven't changed the container, you can start it with the faster:

docker compose up

Moving from Dockerfile to apko for Declarative Images

Thu, 14 May 2026 09:50:03 +0000

Instead of a Dockerfile, we write YAML-based files for apko. This file tells apko exactly which repositories to trust and which packages to materialise into the image.

mkdir apko-demo && cd apko-demo && cat <<EOF > wolfi.yaml
contents:
  repositories:
    - https://packages.wolfi.dev/os
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
  packages:
    - wolfi-base
    - python-3.11
    - py3-pip

entrypoint:
  command: /usr/bin/python3

archs:
  - x86_64
  - aarch64
EOF

Why this is different from a Dockerfile:

repositories & keyring: You are explicitly defining the trust chain.
packages: These are pulled directly from Wolfi. There is no apt-get update.
archs: You can tell apko to build for both Intel and Apple Silicon at the same time.

You can simply run the below command to build the container image.
You don't even need the Docker daemon running for this part.

apko build wolfi.yaml my-wolfi-python:latest output.tar

So, what just happened? Well, apko resolved the dependency tree for Python 3.11. It downloaded the .apk files directly. It then proceeded to unpack them into a filesystem and wrapped them in an OCI-compliant .tar file. It then finished the process by generating an SBOM automatically in the same folder. You can then load and run the image with Docker, even though we never used Docker to build it, we can use Docker to run it.

docker load < output.tar
docker run --rm my-wolfi-python:latest-arm64 --version

Single-layer images are really, really cool

In a standard Dockerfile, every RUN, COPY, and ADD creates a new layer on disk. In apko, everything is flattened into a single, optimised layer. This makes pulling the image across a network much faster because the container runtime only has to fetch and extract one single blob.

docker history my-wolfi-python:latest-arm64

Look in your folder for a file ending in .sbom.json. Open it up. You should have a complete, machine-readable list of every single library inside your image, generated for free.

jq . sbom-aarch64.spdx.json

{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "sbom-sha256:4df02c70193790be7e47ca9943cdce69959308bf13e5b095dc401cb0ae9d352f",
  "spdxVersion": "SPDX-2.3",
  "creationInfo": {
    "created": "2026-05-13T20:49:35Z",
    "creators": [
      "Tool: apko (1.2.12)",
      "Organization: Chainguard, Inc"
    ],
    "licenseListVersion": "3.27"
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "https://spdx.org/spdxdocs/apko/",
  "documentDescribes": [
    "SPDXRef-Package-Image-sha256-7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-Image-sha256-7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf",
      "name": "sha256:7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf",
      "versionInfo": "sha256:7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf",
      "filesAnalyzed": false,
      "description": "apko container image",
      "downloadLocation": "NOASSERTION",
      "supplier": "Organization: Wolfi",
      "primaryPackagePurpose": "CONTAINER",
      "checksums": [
        {
          "algorithm": "SHA256",
          "checksumValue": "7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf"
        }
      ],
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:oci/my-wolfi-python@sha256:7b4c80050663e9a70af6841bbf136a9e4af2d4f3338c1c2a462c8def09ef6cbf?arch=arm64&mediaType=application%2Fvnd.oci.image.manifest.v1%2Bjson&os=linux",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-ImageLayer-sha256-4df02c70193790be7e47ca9943cdce69959308bf13e5b095dc401cb0ae9d352f",
      "name": "sha256:4df02c70193790be7e47ca9943cdce69959308bf13e5b095dc401cb0ae9d352f",
      "versionInfo": "20230201",
      "filesAnalyzed": false,
      "description": "apko operating system layer",
      "downloadLocation": "NOASSERTION",
      "supplier": "Organization: Wolfi",
      "primaryPackagePurpose": "CONTAINER",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:oci/my-wolfi-python@sha256:4df02c70193790be7e47ca9943cdce69959308bf13e5b095dc401cb0ae9d352f?arch=arm64&mediaType=application%2Fvnd.oci.image.layer.v1.tar%2Bgzip&os=linux",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-OperatingSystem-wolfi",
      "name": "wolfi",
      "versionInfo": "20230201",
      "filesAnalyzed": false,
      "description": "Operating System",
      "downloadLocation": "NOASSERTION",
      "supplier": "Organization: Wolfi",
      "primaryPackagePurpose": "OPERATING_SYSTEM"
    },
    {
      "SPDXID": "SPDXRef-Package-apk-wolfi-baselayout-20230201-r29",
      "name": "wolfi-baselayout",
      "versionInfo": "20230201-r29",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "MIT",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "copyrightText": "NOASSERTION",
      "primaryPackagePurpose": "APPLICATION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:apk/wolfi/wolfi-baselayout@20230201-r29?arch=aarch64&distro=wolfi",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-Melange-wolfi-baselayout.yaml-35e7b4713442869116936fae3fe8cf7c69bc9054",
      "name": "wolfi-baselayout.yaml",
      "versionInfo": "35e7b4713442869116936fae3fe8cf7c69bc9054",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "primaryPackagePurpose": "INSTALL",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:github/chainguard-dev/stereo@35e7b4713442869116936fae3fe8cf7c69bc9054#wolfi-baselayout.yaml",
          "referenceType": "purl"
        }
      ]
    },

The above SBOM contains a smoking gun that proves exactly why the Wolfi ecosystem is more secure than standard Docker builds. While a normal SBOM (like one generated by scanning a Debian image) just says, "I think I found a package called X", this Wolfi SBOM includes a specific record that links the final binary back to the original source code instructions. Look at this exact package entry:

Melange is the secret sauce

"SPDXID": "SPDXRef-Package-Melange-wolfi-baselayout.yaml-35e7b4713442869116936fae3fe8cf7c69bc9054",
"name": "wolfi-baselayout.yaml",
"primaryPackagePurpose": "INSTALL",
"referenceLocator": "pkg:github/chainguard-dev/stereo@35e7b4713442869116936fae3fe8cf7c69bc9054#wolfi-baselayout.yaml"

This provides direct software provenance. Most SBOMs stop at the package-level. This SBOM goes a level deeper. It tells you exactly which Melange YAML file was used to build that specific .apk. It’s not just baselayout, it’s basically showing "baselayout as defined in this specific Git commit on GitHub".

In this scenario there’s no guesswork. Most SBOM tools (like Syft or Trivy) have to scan a finished image and guess what’s inside. Because apko builds the image by assembling packages it already knows about, this SBOM is authoritative. It was created during the build, not after. You can also see a reference to chainguard-dev/stereo. This is an internal-facing look at how Wolfi packages are organised. It shows that even the most basic parts of your OS (the baselayout) are treated like verifiable source code.

You’ll also notice the "licenseConcluded": "NOASSERTION" field in some spots. This isn't an error; it's actually a sign of a clean room build. It means apko is strictly reporting what was declared in the package metadata rather than trying to guess the license by regex-scanning files, which is how many security tools end up with false positives.

In short, this SBOM provides a clear chain of custody from your running container all the way back to a specific line of YAML in a GitHub repo. A standard Dockerfile can almost never provide that level of transparency - and that’s why I’m moving over to apko going forward.

Cleanup

As always, I leave a little cleanup script at the end of the exercise:

rm -r apko-demo
docker stop $(docker ps -q)
docker rm $(docker ps -aq)
docker rmi $(docker images -q)
docker images

Why I’m building my Images using Wolfi

Wed, 13 May 2026 15:02:58 +0000

I’m working in software supply chain security, with a specific focus on secure Kubernetes deployments. In a DevOps capacity, I’m focused specifically on ensuring that my container images have as few software vulnerabilities as possible. I believe Wolfi is the best free and open source way to achieve this. This blog post should hopefully demonstrate the value proposition of the Wolfi project.

Getting started with Wolfi

This exercise assumes you already have the Docker runtime installed in order to launch your first Wolfi instance. If docker is already installed, simply open your terminal and pull the base Wolfi image from Chainguard (the creators of Wolfi). This will drop you straight into a Wolfi shell:

docker run -it cgr.dev/chainguard/wolfi-base

At this point, docker downloaded the minimal Wolfi image and started an interactive session. You are now inside the Wolfi OS container. Wolfi uses the apk package manager (same as Alpine Linux), but it uses glibc for better compatibility with modern software. So, within your wolfi shell session, you can simply run the command:

apk update

You'll often hear that wolfi is "distroless-adjacent", or similar phrasing. In short, this means that the distro starts almost empty. If you want to use it for Python development specifically, as an example, you could run the below commands:

apk add python-3.11
python3 --version

If you want to leave the container, simply run the exit command. Because of the --rm flag (if you use it), the container will disappear, keeping your Macbook clean. While this process can be achieved manually by shelling into running containers, in reality, this is now how we would use Wolfi for Projects. Instead, the real power of Wolfi on a Mac is building ultra-secure, tiny images for your own apps, and to be created using a Dockerfile like the example below:

mkdir wolfi-app && cd wolfi-app && cat <<EOF > Dockerfile
FROM cgr.dev/chainguard/wolfi-base
# Install only what you need
RUN apk add nodejs-20
WORKDIR /app
COPY . .
CMD ["node", "index.js"]
EOF

This Dockerfile has a line that says COPY . . and CMD ["node", "index.js"]. Yet, we haven’t created those files yet. This Dockerfile expects a JavaScript file to exist, so let's create a quick one in order to prevent the build from failing:

echo "console.log('Hello from Wolfi running on my Mac!');" > index.js

Now, tell Docker to read your Dockerfile and package it up into an image named my-wolfi-app

docker build -t my-wolfi-app .

The -t command tags / names your image so you don't have to remember a random container ID number. Similarly, the (.) simply tells Docker to look in the current folder for the Dockerfile. Once the build is created, you can run your custom Wolfi container:

docker run my-wolfi-app

So what have we actually done so far? We got a boring helloworld style message from JavScript. Why does that matter? Well, by creating that Dockerfile, we have created a reproducible environment.

From a security standpoint, unlike a standard Mac environment, this container only has nodejs and the bare essentials required to run that helloworld-style application. We’re here to reduce the software dependency bloat, ultimately reducing the overall software supply chain risk surface.
From a portability perspective, you could send that Dockerfile and index.js to a friend, and they would run the exact same environment with the exact same version of Node, regardless of what they have installed on their machine. This needs to work globally, not just on my machine.

If you, like myself, are on any of the M1, M2, or M3 Apple Silicon chips, Wolfi is fantastic because it has native aarch64 support. This means your containers will run at native speeds without needing Rosetta emulation.

Wait, why use this over Ubuntu or Alpine?

As we mentioned from a security standpoint, most standard distros often carry bloat (extra libraries and dependencies) that hackers can exploit. Wolfi is stripped down to the bare essentials and is updated almost daily to fix vulnerabilities (CVEs) faster than almost any other OS. Saying that, when we built our generic Node.js web app, we introduced a lot of software dependencies into the build process. The container ended up being 192 MB in total size.

docker images | grep my-wolfi-app

Comparing Wolfi build sizes versus standard distros

Comparing image sizes is one of the best ways to show off why developers are flocking to Wolfi. It’s significantly leaner than traditional, thick distributions like Ubuntu (39.6 MB), but maintains the compatibility that sometimes breaks in Alpine (3.6 MB).

I’m running all the steps below on my Macbook to pull a few standard images. First of all, Wolfi is a distroless / (un)distro setup, so Wolfi will work the same on Mac as it will work on any other underlying distribution. So let’s use Docker to download the images and use the docker images command to compare them.

docker pull ubuntu:latest
docker pull debian:stable-slim
docker pull alpine:latest
docker images | grep -E "wolfi|ubuntu|debian|alpine"

What you’ll likely see:

Debian Slim: 141 MB
Ubuntu: 180 MB
Wolfi (base): 21.9 MB
Alpine: 13.6MB

The argument here is that Wolfi is nearly as small as Alpine (the gold standard for tiny images), but unlike Alpine, it uses glibc, meaning your real world apps won't break due to missing library links.

Understand whether a container is built using Wolfi

Since Wolfi doesn't have a flashy desktop wallpaper, you have to look at the system identity files and the unique way it handles libraries. By using docker run instead of exec. This creates a container instance from your image, runs the command, and then shuts down.

docker run --rm my-wolfi-app cat /etc/os-release

Checking os-release is likely the most direct proof of wolfi being used to build the container image. Every Linux distribution has a file that describes what it is. The /etc/os-release path is a standard config file in Linux systems (introduced by systemd) that contains operating system identification data, such as the name, version, and ID of the distribution. It is used for identifying the system in scripts, ensuring compatibility, and providing vendor-independent information.

ID=wolfi
NAME="Wolfi"
PRETTY_NAME="Wolfi"
VERSION_ID="20230201"
HOME_URL="https://wolfi.dev"
BUG_REPORT_URL="https://github.com/wolfi-dev/os/issues"

This is the official birth certificate of WolfiOS. You can also check the apk package manager to show your friends where the software actually comes from.

docker run --rm my-wolfi-app cat /etc/apk/repositories

If you want to get really technical, like-for-like comparison of container images to show your friends how many extra (and in some cases unnecessary) tools are sitting inside the general purpose image that a hacker could use - versus Wolfi - just run the below commands:

docker run --rm python:latest ls /usr/bin/ | wc -l
docker run --rm --entrypoint sh cgr.dev/chainguard/python:latest-dev -c "ls /usr/bin/" | wc -l

The below numbers perfectly illustrate why hardened, minimal images like those from Chainguard’s Wolfi are gaining so much traction. The delta we're seeing (655 vs. 326) represents the attack surface. In the official Python image, you have over 300 additional binaries (things like sed, grep, find, and various networking tools) that aren't strictly necessary for your Python code to execute, but provide more living off the land (LOTL-style) opportunities for an attacker.

So, why are the numbers so different?

The official python:latest (655 files) image is based on Debian. It includes a full suite of GNU Coreutils, multiple shells, and legacy system utilities. It’s designed for convenience so that almost any script you write "just works" because all the standard Linux tools are there.
Chainguard’s python:latest-dev (326 files) is alternatively based on Wolfi. Even in the -dev version (which intentionally includes more tools than the production version), they have stripped out almost everything that isn't required to build or run common Python applications.

The hidden reality here is that if developers were to run that same count on the non-dev Chainguard image (the production version), the number would drop even more drastically - likely down to under 10 files. There is essentially only python and the bare-minimum shared libraries (libc, and stuff like that) the basic packages needed to keep the Python interpreter alive in the Chainguard production version.

In a production Chainguard Python image:

There is no ls.
There is no sh.
There is no pip.

Where the developer experience gains are clear for the official python image, since it includes everything so nothing breaks in Day 2 operations, the free-to-use Chainguard dev images provide the hardened development environment required for my builds so it only includes what's needed to build and test the web application. Going further with the Chainguard licensed images, which were never the focus of this blog post, developers receive maximum security, since it includes no shell, no tools, and no unnecessary binaries. You can do additional testing by using cli tools like pip to list python-specific packages in each image you test.

docker run --rm python:latest pip list
docker run --rm cgr.dev/chainguard/python:latest-dev -m pip list

I hope you found this blog useful.

Cleanup

If you want to delete all the locally stored images that aren't currently being used by a container, use the prune command or just remove the images manually with rmi. I’m lazy, so I’m going to just do a nuclear cleanup on everything created during this exercise:

docker stop $(docker ps -q)
docker rm $(docker ps -aq)
docker rmi $(docker images -q)
docker images

Infrastructure as Code: Deploying a Financial Ecosystem with Docker Compose

Fri, 01 May 2026 13:51:55 +0000

Hey folks!

In the previous post, I introduced the overview of My Broker B3. Today we'll talk about the foundation that supports the entire ecosystem: the infrastructure.

For a microservices system of this complexity, manually configuring each database, message broker and monitoring tool would be unmanageable. The solution was to use Docker Compose to create a reproducible local environment that faithfully replicates the needs of a distributed system.

️ The Infrastructure Map

The docker-compose.yml orchestrates 12 containers organized in 5 layers:

┌─────────────────────────────────────────────┐
│           RELATIONAL LAYER (SQL)            │
│  identity-db  wallet-db  order-db  asset-db │
│     MySQL        MySQL    MySQL     MySQL   │
│                  b3-core-db (PostgreSQL)    │
├─────────────────────────────────────────────┤
│              NOSQL LAYER                    │
│           broker-mongodb (Mongo 6.0)        │
├─────────────────────────────────────────────┤
│              CACHE LAYER                    │
│    broker-asset-cache    b3-market-cache    │
│         Redis Alpine          Redis Alpine  │
├─────────────────────────────────────────────┤
│             MESSAGING LAYER                 │
│    kafka (KRaft)        rabbitmq (AMQP)     │
├─────────────────────────────────────────────┤
│           OBSERVABILITY LAYER               │
│        prometheus            grafana        │
└─────────────────────────────────────────────┘

️ Design Decision: Domain Isolation

The most important infrastructure decision was data isolation by domain. Instead of a monolithic database, each microservice has its own instance:

# Each service has its own database — failures are isolated
broker-identity-db:
  image: mysql:8.0
  ports:
    - '3306:3306'  # identity

broker-wallet-db:
  image: mysql:8.0
  ports:
    - '3307:3306'  # wallet

broker-order-db:
  image: mysql:8.0
  ports:
    - '3308:3306'  # orders

broker-asset-db:
  image: mysql:8.0
  ports:
    - '3309:3306'  # assets

b3-core-db:
  image: postgres:15
  ports:
    - '5432:5432'  # B3 core

Why does this matter? If the orders database has a problem, the identity service keeps running. Failures stay isolated in their domain.

Cache: Isolated Instances by Context

Two Redis instances with distinct purposes:

broker-asset-cache:
  image: redis:7.0-alpine
  ports:
    - '${REDIS_BROKER_ASSET_PORT}:6379'  # prices for the broker

b3-market-cache:
  image: redis:7.0-alpine
  ports:
    - '${REDIS_B3_MARKET_PORT}:6379'    # prices for the matching engine

The b3-market-sync-api writes to b3-market-cache and the b3-matching-engine-api reads from it to decide whether to execute or reject an order. The separation ensures that broker-side problems don't affect the B3 side and vice versa.

Messaging: Two Brokers, Two Purposes

Apache Kafka — Internal Event Bus

Configured in modern KRaft mode (no Zookeeper), lighter and more stable for local development:

kafka:
  image: apache/kafka:3.7.0
  environment:
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9092,INTERNAL://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
    KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:9092,INTERNAL://finance-kafka:29092
    KAFKA_CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

Used for the ecosystem's internal topics:

assets-market-data-v1 — asset quotes
order-events-v1 — order lifecycle events

RabbitMQ — Broker ↔ B3 Integration

rabbitmq:
  image: rabbitmq:3-management
  ports:
    - '5672:5672'    # AMQP
    - '15672:15672'  # Management UI

Used exclusively for communication between the broker and the B3 simulator:

mq-broker-to-b3 — orders sent to the matching engine
mq-b3-to-broker — execution feedback

Why two messaging systems? Kafka is ideal for event streams with replay and multiple consumers. RabbitMQ is ideal for work queues with point-to-point delivery guarantees. Each where it makes the most sense.

Observability from Day One

prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    - '9090:9090'

grafana:
  image: grafana/grafana:latest
  ports:
    - '3000:3000'

All Spring Boot microservices export metrics via /actuator/health and /actuator/prometheus. Prometheus collects, Grafana visualizes. From the very first service that starts, we already have observability.

Security: No Hardcoded Secrets

All sensitive configuration lives in a .env file (ignored by Git):

# docker-compose.yml — references only
broker-identity-db:
  environment:
    MYSQL_DATABASE: ${BROKER_IDENTITY_DB_NAME}
    MYSQL_USER: ${BROKER_DB_USER}
    MYSQL_PASSWORD: ${BROKER_DB_PASS}
    MYSQL_ROOT_PASSWORD: ${BROKER_DB_ROOT_PASS}

A .env.example is committed as a template, but actual values never enter the repository.

How to Run

cp .env.example .env
# fill in the variables in .env

docker-compose up -d

A single command brings up all 12 containers. The development environment is identical for anyone who clones the repository.

What's Next?

With the infrastructure running, the next post shows the first microservice: broker-market-data-api — a Python service that fetches real quotes from Brapi, persists them in MongoDB and publishes to Kafka.

About the Series

⬅️ Previous Post: Building a Microservices Ecosystem

➡️ Next Post: Market Data Integrator: Consuming Real-Time Data with Python, MongoDB and Kafka

Series Index: Series Roadmap

Links:

Building Production-Grade Microservices on AWS ECS Fargate with GitLab CI/CD Automation

Wed, 08 Apr 2026 11:03:05 +0000

1. Introduction

Nowadays, applications are moving towards microservices because of their scalability, rollback deployment, and faster development.
But with the growing number of services, deployment pipelines have become:

Slow (like updating all services on every commit)
Hard to manage (multiple pipelines for a single service)
This was causing problems (manual deployments, Docker images being inconsistent, etc.).

I combined AWS Elastic Container Service (ECS) Fargate and Amazon ECR, combined with GitLab CI/CD, which made the pipeline deploy only the service which was changed.

In this guide, I have created an actual production-ready microservices deployment system with:

ECS Fargate for serverless containers
Five microservices:
billing, client_app_api, driver_api, odmu_auth, odmu_contractual
Amazon ECR as an image registry for containers
ALB (Application Load Balancer) with multiple target groups
GitLab CI/CD with folder based triggers
Terraform for infrastructure as code
CloudWatch for monitoring

By the end, you’ll have:

Independent deployments per microservice
Folder-based Automated CI/CD
Production grade ECS services
Scalable, secure, monitored architecture
No manual SSH or PM2

The Real Problem

When I was previously working with ECS, every change in code triggered a full redeployment of all services, although the change was in one service folder. It caused unnecessary downtime and more deployment times, and rollback was risky. I also faced Docker image drift in various environments.

I found a solution

Only the service that changes is rebuilt and deployed
Each service remains fully isolated
Deployments are repeatable and predictable

2. ARCHITECTURE:

Key Components

Component	Purpose
ECS Fargate	Serverless containers (no EC2 management)
ECR	Private container registry
ALB	Routes incoming traffic to the right microservice
CloudWatch	Logs and metrics
Terraform	Infrastructure provisioning
GitLab CI/CD	Automated container build and deployment

Why ECS Fargate?

No EC2 provisioning or patching
Auto-scaling built-in
Secure
Suits for microservices

3. Microservices Structure

Repo Structure:

services/
 ├── billing/
 ├── client_app_api/
 ├── driver_api/
 ├── odmu_auth/
 └── odmu_contractual/
.gitlab-ci.yml
terraform/

Each microservice has:

Its own Dockerfile
Its own business logic
Its own task definition
Its own ECS service

4. Dockerizing the Microservices

Example Dockerfile for client_app_api:

FROM node:20-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .
EXPOSE 3000

CMD ["node", "server.js"]

I used different ports for each microservice:

Service              Port
billing              3001
client_app_api       3000
driver_api           3002
odmu_auth            3003
odmu_contractual     3004

5. Creating ECR Repositories

Account ID: 745812456855
Region: us-east-1

aws ecr create-repository --repository-name billing
aws ecr create-repository --repository-name client_app_api
aws ecr create-repository --repository-name driver_api
aws ecr create-repository --repository-name odmu_auth
aws ecr create-repository --repository-name odmu_contractual

Build and Push (example: billing)

docker build -t billing ./services/billing

docker tag billing:latest \
  745812456855.dkr.ecr.us-east-1.amazonaws.com/billing:latest

aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS \
  --password-stdin 745812456855.dkr.ecr.us-east-1.amazonaws.com

docker push \
  745812456855.dkr.ecr.us-east-1.amazonaws.com/billing:latest

6. ECS Fargate: Task Definition (Terraform)

Example for the billing microservice:

resource "aws_ecs_task_definition" "billing" {
  family                   = "billing-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "512"
  memory                   = "1024"

  execution_role_arn = "arn:aws:iam::745812456855:role/ecsTaskExecutionRole"

  container_definitions = jsonencode([
    {
      name  = "billing"
      image = "745812456855.dkr.ecr.us-east-1.amazonaws.com/billing:latest"

      portMappings = [{
        containerPort = 3001
        protocol      = "tcp"
      }]

      essential = true
    }
  ])
}

Similar type for all microservices.

7. Application Load Balancer Routing

Each microservice has a separate target group:

Service	Target Group Name	Path Pattern
billing	tg-billing	/billing/*
client_app_api	tg-client	/client/*
driver_api	tg-driver	/driver/*
odmu_auth	tg-auth	/auth/*
odmu_contractual	tg-contract	/contract/*

Security Considerations

Security was a first-class concern in this design.

Each ECS task runs with VPC networking. Each ECS task has ENI-level network isolation.
Containers cannot be accessed via SSH; therefore, all deployments take place through CI/CD.
IAM task execution roles are created following the principle of least privilege.
All container images are stored in private ECR repositories on Amazon.
The ALB serves as the only public access point to ECS, and all services remain in a private subnet inside the VPC.

Terraform example:

resource "aws_lb_target_group" "billing" {
  name        = "tg-billing"
  port        = 3001
  protocol    = "HTTP"
  target_type = "ip"
  vpc_id      = var.vpc_id
}

GitLab CI/CD: Folder-Based Deployments
This is the most important section.
Instead of running one giant pipeline or manually using SSH/PM2, I configured the GitLab yml file to only deploy the service that changed.

stages:
  - build
  - deploy

# ----------------------------
# Billing Microservice
# ----------------------------
build_billing:
  stage: build
  only:
    changes:
      - services/billing/**
  script:
    - docker build -t billing ./services/billing
    - docker tag billing:latest 745812456855.dkr.ecr.us-east-1.amazonaws.com/billing:latest
    - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 745812456855.dkr.ecr.us-east-1.amazonaws.com
    - docker push 745812456855.dkr.ecr.us-east-1.amazonaws.com/billing:latest

deploy_billing:
  stage: deploy
  only:
    changes:
      - services/billing/**
  script:
    - aws ecs update-service \
        --cluster production-cluster \
        --service billing-service \
        --force-new-deployment

9. Monitoring & Observability (CloudWatch)

Logs appear under:

/ecs/billing
/ecs/client_app_api
/ecs/driver_api
/ecs/odmu_auth
/ecs/odmu_contractual

Create log groups:

aws logs create-log-group --log-group-name /ecs/billing
aws logs put-retention-policy --log-group-name /ecs/billing --retention-in-days 7

CPU Alarm:

aws cloudwatch put-metric-alarm \
--alarm-name BillingCPUHigh \
--metric-name CPUUtilization \
--namespace AWS/ECS \
--statistic Average \
--period 300 \
--threshold 80 \
--comparison-operator GreaterThanThreshold \
--dimensions Name=ServiceName,Value=billing-service \
--evaluation-periods 2 \
--alarm-actions arn:aws:sns:us-east-1:745812456855:NotifyTeam

Effects and Outcomes of Implementing New Architecture

By deploying this architecture I was able to:

Reduce deployment time, since only those services that had their code changed must be rebuilt.
Allow teams to deploy their microservices independently of one another.
No need to deploy via SSH using PM2.
Create a more predictable and safer rollback process.
Provide for reuse of the same Terraform modules in all target environments.

With the increasing number of services, it became clear that this architecture can be scaled successfully.

10. Lessons Learned:

Here are some of the lessons I learned from building this system:
✔Keep microservices fully isolated
The Docker images, task definitions and ALB routes need to be decoupled.
✔ Use folder-based CI/CD
It reduces deployment times so much.
✔ Choose ECR instead of using public registries
It works well with ECS.
✔ Allocate the right CPU & Memory
Under-provisioning leads to throttling.
Over-provisioning wastes money.
✔ Terraform modules save weeks
When multiple environment setups are in question, reusability is the key.
✔ With this CloudWatch setu you get 80% of your monitoring
Later, add Prometheus or Grafana Cloud.

11. Conclusion

Containers are the foundation of modern infrastructure, and AWS ECS Fargate enables you to run microservices without managing servers. Mixed with GitLab CI/CD and Terraform, you have a scalable, automated, production-grade workflow.
This architecture ensures:

Fast, independent deployments
Secure and isolated workloads
ALB with Fargate for high availability
Complete observability via CloudWatch
Infrastructure is defined as code

If you are deploying microservices on AWS and experiencing slow or high risk deployments, this might be a good working solution validated across real-world environments with the increasing number of services.

If you enjoyed this post, please share it on your favourite social network and get in touch if I can help with anything. Feel free to reach out if you’d like to chat about AWS, DevOps, or container architectures.

Connect with me

• Portfolio: https://khanalsamir.com
• GitHub: https://github.com/Shawmeer
• LinkedIn: https://linkedin.com/in/samir-khanal

If you're working on similar DevOps or AWS projects, feel free to connect. I regularly share practical cloud and CI/CD content.

From image to HTTPS endpoint in one step with ECS Express Mode

Thu, 19 Mar 2026 11:29:16 +0000

Amazon ECS: From EC2 Managed to Express Mode

Amazon ECS has evolved significantly over the years—from managing EC2 container instances manually, to the introduction of Fargate for serverless containers, making infrastructure management seamless. For developers or anyone new to containers, the priority is deploying applications at pace without complexity.

Recently, AWS introduced ECS Managed Instances to simplify launching workloads further. Now the big leap is ECS Express Mode.

What is ECS Express Mode?

Express Mode lets you deploy containerized applications in a single step. Upload your container image from Amazon ECR, optionally specify basic resources like vCPU and memory, or skip resource input to use defaults. Express Mode automatically provisions all necessary AWS infrastructure for you:

Application Load Balancer (ALB)
CloudWatch Logs
Networking & VPC configuration
Auto scaling

This takes away the heavy lifting and infrastructure setup that often slows down deployment.

Why ECS Express Mode Matters

Rapid Deployment: From image to HTTPS endpoint instantly, bypassing manual setup.
Simplified Configurations: Minimal inputs needed. AWS sets up production-ready infrastructure.
Built for Developers: Especially friendly for those new to containers or needing speedy iterations.
Fully Managed: Auto scaling, logging, monitoring, and security come out of the box.

How I Used ECS Express Mode

Provide the container image URI from ECR.

Create or confirm the required IAM roles (task execution and infrastructure). The console can generate these automatically.

Launch the deployment and get an auto-managed HTTPS endpoint with ALB in minutes.

We can what are resources created for us in the timeline view

We can see what are resources created for us in table view

we can also update the express service setup

By Default our deployment stratgey set to canary

Check your website is up or not

It is created everything for me , it is so cool.

Just one click , kaboom everything can be easily destroyed .

Lets now explore the additional configuration option

We can select where this express mode task needs to be scheduled like in which cluster , you can take care about containerPort and health check.
You can see there is env variables options where you can pass you variables from your secrets manager or ssm parameter store

We can pass our Docker CMD commands here . In compute options we can able to set how many vCpus we need and memory . Task scaling etc

we can customize our networking like if you have already available VPC, SG etc . if we dont enable , express mode creates them for us by default

we name our cloudwatchlog group name and prefix , if not it will generated automatically

lets deploy it

IAM Roles Needed

Task Execution Role: Allows ECS to pull images, write logs, and retrieve secrets.
Infrastructure Role: Lets ECS manage ALB, networking, and scaling.

AWS can auto-create these roles with recommended permissions, easing onboarding for new users.

Ending Note

ECS Express Mode represents a huge win for developers and teams who want to focus on building applications, not wrestling with infrastructure. It delivers container deployments at full pace—bringing production-ready, scalable applications live with minimal setup and maximum developer productivity.

Whether you're prototyping or launching at scale, Express Mode simplifies the journey from container image to a fully managed HTTPS endpoint. This is the future of container deployment on AWS ECS—fast, secure, and developer-friendly.

Github Url : https://github.com/aquavis12/ecs-ecr-demo-2025-expressmode

Deploy highly available, scalable containerized applications with simplified configurations. ECS Express Mode takes care of the rest.

Leveraging Laravel with Docker for Streamlined Development

Mon, 09 Mar 2026 19:44:13 +0000

Explore how to integrate Docker into your Laravel development workflow for enhanced efficiency and consistency.

Setting up a development environment can be challenging. Using Docker with Laravel can simplify this process remarkably. Here’s how to get started:

Install Docker: Ensure you have Docker installed on your machine. Follow the official Docker documentation for your operating system.

Create a Dockerfile: In your Laravel project, create a Dockerfile to define your application environment. For example:

FROM php:8.0-fpm
WORKDIR /var/www
COPY . .
RUN apt-get update && apt-get install -y libpng-dev libjpeg-dev libfreetype6-dev && docker-php-ext-configure gd --with-freetype --with-jpeg && docker-php-ext-install gd
CMD php artisan serve --host=0.0.0.0

This example sets up PHP with GD support, essential for image processing in Laravel applications.

Create a docker-compose.yml file: This file will manage your multi-container application. For Laravel, it might look like this:

version: "3.8"
services:
  app:
    build:
      context: .
    volumes:
      - .:/var/www
    ports:
      - "8000:8000"
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: laravel

The above example sets up a MySQL database alongside your Laravel app.

Run the containers: Execute the following command in your terminal:
```
docker-compose up -d
```
This command will bring up your Laravel application and your database as defined in the docker-compose.yml file.
Access your application: You can now access your Laravel application running on http://localhost:8000.

Final Thoughts: By using Docker, you ensure a consistent development environment across different setups, leading to fewer deployment headaches. It’s a game-changer for collaborative projects where each developer may have different local setups.

Take Action: Try integrating Docker into your Laravel project today and notice how it changes your workflow for the better.

Coder Legion - Recent posts tagged containers

You patched PHP. The old image is still in your cluster’s future.

tags: kubernetes, security, php, devops

How most teams found out

How fast the scanners caught up

Where the scanners stay blind

Patching is half the job

What this looks like with an admission controller in the path

The checklist, tool-agnostic

Closing

How and Why I Built My Own bootc-Based Linux Distribution

How I Installed It

Developing the System from Within

DisplayLink, Because of Course It Couldn't Be That Easy

Updates Managed by CI/CD

Summary

One File, Six Services: Running AI Agents With Docker Compose

An Agent Is Never One Process

Three Patterns, Three Cost Profiles

The Parts That Bite Later

Where Compose Stops Being The Answer

Understanding the vectors of container escapes

How isolated runtimes can mitigate these risks

Lateral credential harvesting via process namespace sharing

Example script

What is the security misconfiguration exactly?

How it works technically

Strategy 1 - PSS in Kubernetes

Strategy 2 - Runtime Isolation

Breakouts, Bots, and Bare Metal

What is T1611?

When code execution becomes autonomous

M1048 - Application isolation and sandboxing

Closing the door on sandbox escapes

From Dockerfile to Docker Hub: A Complete Beginner's Guide to Docker

Docker Workflow Overview

What Are We Building?

Project Structure

Step 1: Create the Node.js Application

app.js

package.json

Application Architecture

Step 2: Create the Dockerfile

Understanding the Dockerfile

Base Image

Working Directory

Copy Dependency Files

Install Dependencies

Copy Application Source Code

Expose Application Port

Start the Application

What Happens During Docker Build?

Step 3: Build the Docker Image

Step 4: Verify the Image

Docker Image vs Docker Container

Step 5: Run a Container

Understanding the Command

Detached Mode

Container Name

Port Mapping

Step 6: Verify the Container

Step 7: Access the Application

Step 8: Login to Docker Hub

Why Push Images to Docker Hub?

Step 9: Push the Image

Step 10: Verify on Docker Hub

Real-World DevOps Workflow

Key Takeaways

Conclusion

Running Astro in a preview container

Add a Docker compose file

The Docker file

The Vite config change

Spinning it up

Moving from Dockerfile to apko for Declarative Images

Why this is different from a Dockerfile:

Single-layer images are really, really cool

Melange is the secret sauce

Cleanup

Why I’m building my Images using Wolfi