Coder Legion - Recent posts tagged kubernetescluster

How a Kubernetes Cluster Works | Control Plane & Worker Nodes

Tue, 29 Sep 2026 12:00:00 +0000

How does a Kubernetes cluster actually work?

I’ve been learning Kubernetes and realized that it’s easy to memorize terms like Control Plane, Worker Node, API Server, Scheduler, and kubelet without really understanding how they fit together.

So I put together a video explaining how a Kubernetes cluster works, including:

Control Plane and Worker Nodes
API Server
etcd
Scheduler
Controller Manager
Pods
kubelet
Container Runtime
kube-proxy
What happens when you run a kubectl command?
Different ways to create and operate a Kubernetes cluster

My goal was to understand the flow, not just memorize component definitions.

🎥 Watch here:
How a Kubernetes Cluster Works | Control Plane & Worker Nodes

I’d love to hear from other Kubernetes learners like me: which part of the cluster architecture took you the longest to understand?

How a Football Match Helped Me Understand Kubernetes: Desired State vs Actual State

Sat, 18 Jul 2026 10:00:00 +0000

Have you ever wondered what Desired State and Actual State mean in Kubernetes?

I used to hear those terms all the time, but they didn't truly make sense until I watched a football match.

In this video, Ada explains one of Kubernetes' most important concepts using a simple football analogy. Instead of complicated diagrams and technical jargon, you'll see how a football coach, substitutions, and an unexpected injury perfectly explain how Kubernetes constantly works to keep your applications running.

Whether you're preparing for the KCNA, CKA, or CKAD, learning Kubernetes for DevOps, or just getting started with cloud-native technologies, this real-world story will help you understand one of Kubernetes' core ideas.

What you'll learn

What Desired State means in Kubernetes
What Actual State means
How Kubernetes compares both states
What reconciliation is
Why Kubernetes automatically creates new Pods
Why this makes Kubernetes reliable

🎯 Watch next

What a Busy Restaurant Can Teach You About Kubernetes Services

🚀 Want to practice Kubernetes yourself?

Spin up a managed Kubernetes cluster on DigitalOcean

You can follow my work and projects on Twitter, LinkedIn, GitHub, and YouTube, where I share what I am building and learning along the way.

If this video helped you understand Kubernetes better, consider subscribing for more real-life cloud and DevOps stories.

Thank you for reading.

ReRute: Treating Physical Freight Like Packets on a Network. Kubernetes and Docker on Logistics

Sun, 28 Jun 2026 14:51:44 +0000

How we combined Go, Elixir, ScyllaDB, and Kotlin to replace manual dispatching with autonomous reconciliation loops and instant smart-contract settlements.

If you have ever peered behind the curtain of modern logistics and freight forwarding, you know a dark secret: global supply chains run on spreadsheets, phone calls, static whiteboards, and net-60 invoicing.

When a shipper needs to move 500 kilograms of temperature-sensitive pharmaceuticals across a city or country, they typically negotiate with a middleman broker. A specific driver and vehicle are manually assigned to a static route. But the physical world is chaotic. Traffic jams happen. Refrigeration units fail. Trucks break down. When an anomaly occurs today, human dispatchers scramble to make frantic phone calls while cargo spoils and drivers wait months for payment.

We built ReRute to solve this foundational friction. Instead of treating logistics as a series of rigid, manual bookings, ReRute introduces a Declarative Physical Asset Orchestration Protocol. We treat physical freight like packets routing dynamically across a digital network.

The Paradigm Shift: Declarative Orchestration
In software engineering, Kubernetes revolutionized infrastructure by popularizing declarative orchestration. You don't tell the server how to start processes step-by-step; you declare a Desired State ("I want 3 replicas of this container running"), and an autonomous controller continuously monitors the Actual State, reconciling the two until they match.

ReRute applies this exact philosophy to physical atoms:

Cargo Pods (Desired State): Shippers use the Client Portal to declare a Cargo Pod Manifest. They don't book a specific driver. They declare requirements: Origin, Destination, Weight ($kg$), Volume ($m^3$), Quote Amount, and Environmental Constraints (e.g., refrigerated: true).

Transport Nodes (Actual State): Drivers operate vehicles equipped with our headless Android Node Agent. The node registers its physical capabilities (max load capacity, refrigeration status) and continuously streams real-time physical telemetry (GPS coordinates, vector speed, heading).
Continuous Reconciliation: A centralized control plane continuously compares pending Cargo Pods against active Transport Nodes, autonomously assigning, routing, and rerouting cargo based on proximity, capacity, and environmental compliance.

The Architecture: Bottom-Up Design

1. Storage Layer: Sub-Millisecond Telemetry with ScyllaDB
Tracking thousands of moving transport nodes generating GPS pings every 10 seconds creates a massive, high-frequency write load. Relational databases choke on this time-series ingestion.

We built our storage foundation on ScyllaDB, a hyper-fast C++ NoSQL database compatible with Apache Cassandra. Using partitioned time-window compaction strategies (TimeWindowCompactionStrategy), ScyllaDB absorbs millions of spatial-temporal coordinate pings (latitude, longitude, speed_kmh, heading_degrees) with sub-millisecond latency while maintaining absolute cluster state.

2. The Orchestration Brain: Go Control Plane

At the heart of ReRute sits the Control Plane, written in Go for maximum concurrency and minimal memory footprint.

Every five seconds, the Go API runs a background reconciliation loop. It queries ScyllaDB for all unassigned Cargo Pods (Pending_Pickup) and scans the spatial registry for available Transport Nodes (Available) that satisfy all physical constraints. When a match is found, state is atomic-locked and pushed to the driver's device.

More importantly, the Control Plane is self-healing. If an active driver experiences a mechanical failure and taps "Report Breakdown" on their device, the Go reconciliation engine immediately cordons that node (Cordoned), releases the Cargo Pod back into the routing pool, and autonomously re-dispatches the nearest available transport node to perform a mid-route pickup.

3. Financial Rails: Elixir & The Daraja Smart Escrow Engine

In logistics, trust is tied directly to cash flow. Drivers often wait 30 to 90 days to get paid by brokers. ReRute eliminates net-terms entirely using automated digital escrow.

Our Escrow Engine is built in Elixir, leveraging the Erlang BEAM VM for unparalleled fault tolerance and concurrency. When a client initiates a shipment, upfront payment is locked in a stateful liability machine: PENDING_PICKUP ➔ IN_TRANSIT ➔ NODE_LOCKED ➔ SETTLED.

To handle mobile money payments across East Africa, our Elixir engine integrates directly with Safaricom’s Daraja API. Because cellular network latency can cause payment webhooks to duplicate or lag, the engine enforces strict idempotency using Redis and Elixir GenServer supervisors.

The exact second a driver confirms package handover on their mobile device, the physical state matches the desired state. The Elixir state machine instantly transitions to SETTLED, calculates platform margins, and fires an automated B2C payout directly to the driver's M-Pesa mobile wallet. Settlement drops from 60 days to 600 milliseconds.

4. The Physical Edge: Kotlin Node Agent
To ensure reliable telemetry without draining battery life, the mobile edge is built natively in Kotlin. Running as a headless background daemon on Android devices, the Node Agent wakes up on a precise ticker, reads physical sensors (ACCESS_FINE_LOCATION), computes vector trajectories, and transmits lightweight JSON payloads over secure Cloudflare tunnels.

The Future of Autonomous Supply Chains
ReRute demonstrates what is possible when we stop viewing physical logistics as a human-coordination problem and start treating it as a distributed software protocol. By combining ultra-fast time-series databases, concurrent Go reconciliation engines, and fault-tolerant Elixir financial state machines, we create a supply chain that is transparent, autonomous, and self-healing.

Whether moving pharmaceuticals across Nairobi or managing heavy freight across continents, declarative physical orchestration ensures that cargo always finds its optimal route home.
WATCH DEMO VIDEO
WATCH DEMO VIDEO 2

THIS IS STILL A WORK IN PROGRESS BUT IF YOU ARE WILLING TO WORK WITH
ME THEN WE CAN TURN IT INTO A WORKING PRODUCT

Deploying a Simple App on K3S in AWS EC2 with GitHub Actions & ECR

Tue, 24 Mar 2026 10:36:12 +0000

title: Deploying a Simple App on K3S in AWS EC2 with GitHub Actions & ECR
published: true
date: 2025-09-07 12:15:31 UTC
tags: github,kubernetes,kubernetescluster,githubactions

canonical_url: https://medium.com/@mahinshanazeer/deploying-a-simple-app-on-k3s-in-aws-ec2-with-github-actions-ecr-58e0ea29eb2a

In this session, we’ll walk through the configuration of K3S on an EC2 instance and deploy a multi-container application with a frontend, backend, and database. The application will run inside a Kubernetes cluster using Deployments and StatefulSets in headless mode. For the setup, we’ll use EC2 to host the cluster, GitHub as our code repository, and GitHub Actions to implement CI/CD.

If you’re an absolute beginner and not familiar with configuring EC2, I recommend checking out my blog here:

Step-by-Step Guide to Launching an EC2 Instance on AWS : For Beginners

This will be an end-to-end project deployment designed for those learning K3S, CI/CD, and Docker. You’ll gain hands-on experience in setting up CI/CD pipelines, writing Dockerfiles, and using Docker Compose. We’ll then move on to deploying the application in K3S, working with Kubernetes manifests, and exploring key components such as Deployments, Services (NodePort and ClusterIP), ConfigMaps, Persistent Volumes (PV), Persistent Volume Claims (PVC), and StatefulSets.

K3S is a lightweight Kubernetes distribution developed by Rancher (now SUSE). It’s designed to be:

Lightweight — small binary, minimal dependencies.

Easy to install — single command installation.

Optimized for edge, IoT, and small clusters — runs well on low-resource machines like Raspberry Pi or small EC2 instances.

Fully compliant — supports all standard Kubernetes APIs and workloads.

In short, K3S simplifies Kubernetes and makes it resource-efficient, making it ideal for single-node clusters, test environments, and learning purposes.

Log login to the EC2 machine and install k3s first:

K3s

You can install K3S on your machine using the following single command:

sudo apt update -y && sudo apt upgrade -y
curl -sfL https://get.k3s.io | sh - 
# Check for Ready node, takes ~30 seconds 
sudo k3s kubectl get node

Installation of k3s

Once the installation is completed, the output should be similar to this:

Kubectl node status

Once the cluster is up and running, we can move on to the application. You can refer to the following repository for the demo To-Do List app. Before cloning the repository, make sure Docker is installed on the machine to build and test the application. For installing Docker, refer to the following URL:

Ubuntu

#run the following command first to remove conficting packages

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

#Installing Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

#Now verify the installtion.
sudo docker run hello-world

Now, let’s dive into the demo application. The Application Stack:

Frontend: React.js
Backend API: Node.js + Express
Database: MongoDB
Containerization & Registry: Docker + AWS ECR
Orchestration & Service Management: Kubernetes (K3s)

Next, let’s clone the application repository to your local machine.

GitHub - mahinshanazeer/docker-frontend-backend-db-to_do_app: Simple Application with Frontend + Backened + DB

git clone https://github.com/mahinshanazeer/docker-frontend-backend-db-to_do_app

Clone the github application

Once the repository is cloned, switch to the application directory and check for the Docker Compose file.

Directory structure

version: "3.8"
services:
  web:
    build:
      context: ./frontend
      args:
        REACT_APP_API_URL: ${REACT_APP_API_URL}
    depends_on:
      - api
    ports:
      - "3000:80"
    networks:
      - network-backend
    env_file:
      - ./frontend/.env

  api:
    build: ./backend
    depends_on:
      - mongo
    ports:
      - "3001:3001"
    networks: 
      - network-backend
  
  mongo:
    build: ./backend-mongo  
    image: docker-frontend-backend-db-mongo
    restart: always
    volumes: 
      - ./backend-mongo/data:/data/db
    environment: 
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: adminhackp2025
    networks: 
      - network-backend

networks:
  network-backend:

volumes: 
  mongodb_data:

In the Docker Compose file, you’ll see sections for web, api, and mongo. Let’s dive into each directory and review the Dockerfiles. The Docker Compose file builds the Docker images using the Dockerfiles located in their respective directories.


root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/frontend# cd /home/ubuntu/docker-frontend-backend-db-to_do_app/frontend
root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/frontend# cat Dockerfile 
# ---------- Build Stage ----------
FROM node:16-alpine AS build

WORKDIR /app

# Copy dependency files first
COPY package*.json ./

# Install dependencies
RUN npm install --legacy-peer-deps

# Copy rest of the app
COPY . .

# Build the React app
RUN npm run build

# ---------- Production Stage ----------
FROM nginx:alpine

# Copy custom nginx config if you have one
# COPY nginx.conf /etc/nginx/conf.d/default.conf

# Copy build output from build stage
COPY --from=build /app/build /usr/share/nginx/html

# Expose port 80
EXPOSE 80

# Start nginx
CMD ["nginx", "-g", "daemon off;"]

***

root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/backend# cd /home/ubuntu/docker-frontend-backend-db-to_do_app/backend
root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/backend# cat Dockerfile 
FROM node:10-alpine

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 3001

***

root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/backend# cd /home/ubuntu/docker-frontend-backend-db-to_do_app/backend-mongo/
root@ip-172-31-22-24:/home/ubuntu/docker-frontend-backend-db-to_do_app/backend-mongo# cat Dockerfile 
FROM mongo:6.0
EXPOSE 27017

Open the .env file in the frontend directory and update the IP address to your EC2 public IP. This environment variable is used by the frontend to connect to the backend, which runs on port 3001.

vi /home/ubuntu/docker-frontend-backend-db-to_do_app/frontend/.env

#edit the IP address, I have updated my EC2 public IP
~~~
REACT_APP_API_URL=http://54.90.185.176:3001/
~~~

We can also cross-check the total number of APIs using the following commands:

grep -R "router." backend/ | grep "("
grep -R "app." backend/ | grep "("
grep -R "app." backend/ | grep "(" | wc -l
grep -R "router." backend/ | wc -l

Let’s test the application by spinning up the containers. Navigate back to the project’s root directory and run the Docker Compose command.

cd /home/ubuntu/docker-frontend-backend-db-to_do_app
docker compose up -d

Once you run the command, Docker will start building the images and spin up the containers as soon as the images are ready

building docker containers

Wait until you see the ‘built’ and ‘created’ messages. Once the containers are up and running, use docker ps -a to verify the status.

build completed and containers started.

docker ps -a

docker processes

Once the Docker containers are up and running, verify that the application is working as expected.

Once the Docker containers are up and running, verify that the application is working as expected. Open the server’s IP address on port 3000. You can confirm the mapped ports in the Docker Compose file or by checking the docker ps -a output. Here, port 3000 is for the frontend web app, port 3001 is for the backend, and MongoDB runs internally on port 27017 without public access. In this example, load the website by entering 54.90.185.176:3000 in your browser.

Application interface

If you’re using Chrome, right-click anywhere on the page and open Inspect > Network. Then click on Add Todo to verify that the list updates correctly, and the network console shows a 200 status response

checking the network

Application testing

Click on the buttons and try to add new file, and verify the status codes:

Testing

So far, everything looks good. Now, let’s proceed with the Kubernetes deployment. To configure resources in Kubernetes, we’ll need to create manifest files in YAML format. You can create these files as shown below.

mkdir /home/ubuntu/manifest
touch api-deployment.yaml api-service.yaml image_tag.txt mongo-secret.yaml mongo-service.yaml mongo-statefulset-pv-pvc.yaml web-deployment.yaml web-env-configmap.yaml web-service.yaml

Now edit each file and add the following contents:

api-deployment.yaml:

Defines how the backend API should run inside the cluster.

Creates 2 replicas of the API for reliability.

Uses environment variables from secrets for MongoDB authentication.

Ensures the API pods always restart if they fail.

Importance: Provides scalability and fault tolerance for the backend service.

Rolling Update: Gradually replaces old pods with new ones. Uses fewer resources, minimal downtime if tuned, but users may hit bad pods if the new version is faulty.

Rolling = efficient and native.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: 495549341534.dkr.ecr.us-east-1.amazonaws.com/hackp2025:api-20250907111542
          ports:
            - containerPort: 3001
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: username
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: password
      restartPolicy: Always

api-service.yaml

Exposes the API deployment to the outside world.

Type NodePort makes the service reachable via :31001.

Ensures frontend or external clients can communicate with the backend.

Importance: Acts as a bridge between users/frontend and the backend API.

apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app: api
spec:
  type: NodePort
  selector:
    app: api
  ports:
    - port: 3001 # internal cluster port
      targetPort: 3001 # container port
      nodePort: 31001 # external port on the node

mongo-secret.yaml

Stores sensitive information (username & password) in base64-encoded format.

Used by both the API and MongoDB.

Keeps credentials out of plain-text manifests.

Importance: Secure way to handle database credentials.

apiVersion: v1
kind: Secret
metadata:
  name: mongo-secret
type: Opaque
data:
  # Base64 encoded values
  username: YWRtaW4= # "admin"
  password: YWRtaW5oYWNrcDIwMjU= # "adminhackp2025"

mongo-service.yaml

Defines the MongoDB service.

- ClusterIP: None makes it a headless service , required for StatefulSets.

Allows pods to connect to MongoDB by DNS (e.g., mongo-0.mongo).

Importance: Provides stable networking for MongoDB StatefulSet pods.

apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    app: mongo
  clusterIP: None # headless service for StatefulSet

mongo-statefulset-pv-pvc.yaml

Handles the database persistence and StatefulSet definition.

- PersistentVolume (PV): Reserves storage (5Gi).

- PersistentVolumeClaim (PVC): Ensures pods can claim storage.

- StatefulSet: Guarantees stable network identity and persistent storage for MongoDB.

Importance: Ensures MongoDB data is preserved even if the pod restarts.

Blue/Green Deployment: Runs two environments (Blue = live, Green = new). Traffic is switched instantly once Green is ready. Near-zero downtime and easy rollback, but requires double resources and is more complex for stateful apps.

Blue/Green = safer cutover, higher cost.

# PersistentVolume for Green MongoDB
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongo-green-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /root/hackpproject/data-green # separate path for green
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "" # Must match PVC in StatefulSet

---

# StatefulSet for Green MongoDB
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-green
  labels:
    app: mongo
    version: green
spec:
  serviceName: mongo # existing headless service
  replicas: 1
  selector:
    matchLabels:
      app: mongo
      version: green
  template:
    metadata:
      labels:
        app: mongo
        version: green
    spec:
      containers:
        - name: mongo
          image: 495549341534.dkr.ecr.us-east-1.amazonaws.com/hackp2025:db-20250907111542
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: username
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: password
          volumeMounts:
            - name: mongo-data
              mountPath: /data/db
  volumeClaimTemplates:
    - metadata:
        name: mongo-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi
        storageClassName: "" # binds to the pre-created PV

web-deployment.yaml

Defines how the frontend (React.js app) should run.

Runs 2 replicas for high availability.

Pulls API endpoint from ConfigMap.

Resource requests/limits ensure fair scheduling.

Importance: Deploys the UI and links it to the backend API via config.

Rolling Update: Gradually replaces old pods with new ones. Uses fewer resources, minimal downtime if tuned, but users may hit bad pods if the new version is faulty.

Rolling = efficient and native.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3         
      maxUnavailable: 0   
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: 495549341534.dkr.ecr.us-east-1.amazonaws.com/hackp2025:web-20250907111542
          ports:
            - containerPort: 3000
              protocol: TCP
          env:
            - name: REACT_APP_API_URL
              valueFrom:
                configMapKeyRef:
                  name: web-env
                  key: REACT_APP_API_URL
          resources:
            requests:
              cpu: "200m"
              memory: "1024Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
      restartPolicy: Always

web-env-configmap.yaml

Stores non-sensitive environment variables.

Defines the API endpoint for the frontend (REACT_APP_API_URL).

Can be updated easily without rebuilding Docker images.

Importance: Provides flexibility to change configuration without redeploying code.

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-env
  labels:
    app: web
data:
  REACT_APP_API_URL: http://98.86.216.31:31001

web-service.yaml

Exposes the frontend to users.

- Type NodePort makes it available externally at :32000.

Maps port 3000 (service) → port 80 (container).

Importance: Allows end-users to access the web app from their browser.

apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
spec:
  type: NodePort
  selector:
    app: web # Must match Deployment labels
  ports:
    - name: http
      port: 3000 # Service port inside cluster
      targetPort: 80 # Container port
      nodePort: 32000 # External port accessible from outside

We have now moved all the manifest files to /root/hackpproject/manifestfiles.

Once the manifests are finalised, the next step is to create a repository in ECR to push the build artefact images.

Steps to Create an ECR Repository:
1.Log in to AWS Console → Go to the ECR service.

Create Repository

Click Create repository.

Select Private repository.

Enter repository names — prodimage. (In this case, we are creating a single repository for all those 3 images)

Leave others as default and click Create repository.

Authenticate Docker with ECR

Step 1: Finding the ECR

Step 2: Creating Repository

Step 3: onfiguring Repository

Once the registry is created, you can proceed with the CI/CD pipeline.

Reposiroty end point

Now, let’s create a GitHub Actions pipeline to deploy the code to the EC2 K3S cluster. The first step is to configure GitHub Actions with access to the repository, ECR, and the EC2 instance via SSH.

Navigate to the project directory and create a folder named ‘.github’. Inside this folder, create a file named ‘ci-cd.yml’.

mkdir .github
cd .github
touch ci-cd.yml
vi ci-cd.yml

The ci-cd.yml file is the core configuration file for GitHub Actions that defines your CI/CD pipeline. Now use the following script in that ci-cd.yml file:

name: CI/CD Pipeline

on:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on

Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide

Sat, 21 Mar 2026 13:53:24 +0000

title: Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide
published: true
date: 2025-10-16 14:35:59 UTC
tags: raspberrypi,kubernetes,etcd,kubernetescluster

canonical_url: https://medium.com/@mahinshanazeer/upgrading-our-raspberry-pi-kubernetes-cluster-from-v1-29-to-v1-34-etcd-backup-guide-97a2074237b4

Keeping your cluster up-to-date is crucial for security, stability, and access to new features. In this blog, we’ll take a closer look at our home lab Kubernetes cluster running on Raspberry Pi devices. The setup includes 1 master node and 3 worker nodes, and it has been running smoothly for over four months without any major issues.

k8s home lab configuration — nodes

master2 — master node/ control plane
master1 — worker node 3
worker 1 — worker node 1
worker 2 — worker node 2

k8s home lab configuration — system components

As part of regular maintenance and to stay up-to-date with the latest features and security patches, we decided to upgrade the cluster from Kubernetes v1.29 to v1.34. Alongside the upgrade, we’ll also walk through the process of backing up the ETCD datastore, which is crucial for fault tolerance and disaster recovery.

Here’s what we’ll cover:

⬆️ Step-by-step guide to upgrading Kubernetes components.
️ Safely backing up ETCD in the master node.
✅ Post-upgrade verification.

A. Backing Up ETCD

Before diving into the upgrade process, it’s essential to create a backup of ETCD, the key-value store that holds all cluster state data. This step is crucial because if anything goes wrong during the upgrade — whether it’s a misconfiguration or a failed component — we need a reliable way to restore the cluster to its current stable state.

Think of the ETCD backup as your safety net. It ensures that even in the worst-case scenario, you won’t lose your cluster’s configuration, workloads, or networking setup.

Run the following command to confirm the ETCD pod name running on your control plane:

kubectl get pods -n kube-system | grep etcd

k8s configuration — ETCD setup

Suppose your control plane (master node) runs ETCD as a static pod (which is the default in kubeadm-based clusters, including Raspberry Pi setups). In that case, you can run etcdctl directly on the master node host — because the ETCD data directory and certificates are mounted locally at /etc/kubernetes/pki/etcd/ and /var/lib/etcd/. In that case, you can use the following command to take the backup:

sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

If ETCD runs as a container (inside the pod) or you don’t have the etcdctl binary available on the host, then you’ll need to exec into the pod to run the backup from inside the container.

In my setup, where etcd runs as a container, I need to access the pod directly to execute the backup. There is no etcdctl binary available on the host.

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

#edit the <node-label> with etcd node

Creating backup

Now, verify the snapshot using the following command:

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot status /var/lib/etcd/backup.db -w table

In the above screenshot, you can seee a deprecation warning. ETCD v3.6+ is deprecated etcdctl snapshot status and recommends using etcdutl snapshot status. However, in my kubeadm static pod setup, the etcdutl binary is not included in the ETCD container. Only etcdctl exists inside the pod. The warning is simply a deprecation notice, not a failure. You can safely continue using it etcdctl snapshot status — it works perfectly.

In my setup, I came across a deprecation warning while using etcdctl. To verify, I checked the current version of etcd using the following command.

Since my cluster is kubeadm-managed, even though etcd v3.6+ recommends using etcdutl for snapshot management, etcdctl inside the etcd pod still works perfectly for backup and restore. etcdutl is a newer utility introduced in etcd v3.6+ as a replacement for certain snapshot operations, but in my case, continuing with etcdctl is fully sufficient.

kubectl exec -n kube-system etcd-master2 -- etcdctl version

etcd version checking

In some scenarios, the etcd backup file may reside only inside the etcd pod and not in a host-accessible directory.

This can happen if the etcd container is minimal and lacks tools like tar to move files around, or if the host does not have etcdctl installed to create the snapshot directly. In such cases, you can use specific commands to move the backup file to a host-accessible directory.

kubectl cp kube-system/etcd-master2:/tmp/backup.tar.gz ./backup.tar.gz

In my setup, I created the etcd snapshot directly on the host in a host-accessible directory (for example, /var/lib/etcd/backup.db). Since the backup is already on the host, it can be copied or archived without using kubectl cp.

sudo cp /var/lib/etcd/backup.db ~/backup.db
sudo chown $USER:$USER ~/backup.db
ls -l ~/backup.db

backup.db file status

Additionally, it’s a good practice to create a copy of the backup on a remote server. This ensures an extra layer of safety and helps protect against data loss in case of hardware failure or accidental deletion.

B. Upgrading the cluster

Since we have backed up etcd, we can safely proceed with the kubeadm upgrade for our Raspberry Pi Kubernetes cluster. Let’s go through a step-by-step process to upgrade the cluster from v1.29 → v1.34.

Kubernetes officially supports upgrading one minor version at a time, which means that to move from 1.29 to 1.34, you need to perform incremental upgrades:

Step 1: v1.29 → v1.30
Step 2: v1.30 → v1.31
Step 3: v1.31 → v1.32
Step 4: v1.32 → v1.33
Step 5: v1.33 → v1.34

Please refer to the following URL for k8s official documentation on skew policy:

Version Skew Policy

Step 1: Pre-Checks for upgrading the cluster

Before upgrading the cluster, it’s essential to ensure that the cluster is healthy and stable. This helps prevent issues during the upgrade and ensures a smooth transition between versions.

kubectl version
kubectl get nodes
kubectl get cs
kubectl get pods -n kube-system

Current status of the cluster

Step 2: Upgrade kubeadm on the Master Node

You can also refer to the following documentation for more details regarding kubeadm upgrade:

Upgrading kubeadm clusters

To upgrade to v1.30, ensure that the directory /etc/apt/keyrings exists. If it doesn’t, create it before running the curl command (see the note below). Afterwards, update the package repository to target v1.30.

Each Kubernetes minor release repository (v1.29, v1.30, v1.31, etc.) may use a different GPG key. Using the key from an older version, such as v1.29, can cause package verification failures when installing packages from the v1.30 repository.

To prevent such errors, always use the GPG key and repository provided specifically for the minor version you are upgrading to.

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Before upgradingkubeadmEnsure your package lists are up to date. Then, install the required version by using the following commands

sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

If you want to upgrade to a specific Kubernetes version, you can use the following command to list all available minor versions in the corresponding repository:

apt-cache madison kubeadm

available packages in the repo

For comprehensive guidance on upgrading your Kubernetes cluster to version v1.30 , refer to the official Kubernetes documentation:

Installing kubeadm

This resource provides detailed, step-by-step instructions tailored for clusters managed with kubeadm , ensuring a smooth and efficient upgrade process.

Once the upgrade is complete, verify the kubeadm version using the following command:

kubeadm version
kubectl version --client
kubelet --version

current kubeadm version

Step 3: Plan the Upgrade

After upgrading kubeadm, it’s important to check the upgrade path and verify the versions of all cluster components. This ensures that all parts of your Kubernetes cluster are compatible and running the expected versions.

sudo kubeadm upgrade plan

It will display:

The current cluster version and control plane version.
The available target versions you can upgrade to.
The versions of kubelet and kubectl on your nodes.
Any required steps or notes for a smooth upgrade.

kubeadm upgrade plan output — 1

kubeadm upgrade plan output — 2

Step 4: Upgrade the Control Plane

Once the upgrade plan is reviewed and all changes are understood, you can proceed with upgrading the cluster. Use the kubeadm upgrade apply command, which is provided in the output of kubeadm upgrade plan.

sudo kubeadm upgrade apply v1.30.14

The upgrade process may take some time, depending on the size and complexity of your cluster. Once it completes successfully, you should see a message similar to the following:

upgraded to “v1.30.14”

Now, verify the control plane using the following commands:

#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Step 5: Upgrade Worker Nodes

Now repeat the same steps from ‘Step 2: Upgrade kubeadm on the Master Node’ again in master1, worker1 and worker2 nodes (worker nodes). Workers do not control the cluster state; running upgrade plan there doesn’t provide meaningful information. First drain the worker node from the control plane, so that all the resources will be taken out of the node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

draining resources from worker node (master1)

Now you can repeat the steps we followed for control plane for upgrading the version:

ssh to master1 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
kubeadm upgrade node

upgrading worker node — host ‘master2’ is a worker node here. (master2 — master node/ control planemaster1 — worker node 3worker 1 — worker node 1worker 2 — worker node 2)

Also verify the nodes from control plane:

upgrade completed

Now we can following commmand to make the node available for schedule;

kubectl uncordon master1
#run this command in the control plane.

Marking node as schedulable

kubectl uncordon command is used to mark a node as schedulable again after it has been drained.

Similarly we can do the upgrades in other 2 worker nodes, I will mention the commands below:

worker1:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker1 node

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker1
kubectl get nodes
kubectl get pods -n kube-system

worker2:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker2 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker2 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker2

All nodes have now been successfully upgraded to v1.30.14 , as shown in the following screenshot.

kubernetes version v1.30.14

Step 5: Upgrading to 1.34

Following the same process, we’ll now proceed to upgrade the cluster from v1.30.14 to v1.31. Then we will proceed with with

Step 1: v1.29.15 → v1.30.14–1.1
Step 2: v1.30.14–1.1 → v1.31.13–1.1
Step 3: v1.31.13–1.1 → v1.32.9
Step 4: v1.32.9 → v1.33.5
Step 5: v1.33.5 → v1.34.1

In control-plane (master2) use the following steps to upgrade to version 1.31:

ssh to controlplane (master2)

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 2:
sudo apt update
apt-cache madison kubeadm

#step 3:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm kubelet kubectl #will take the latest version
sudo apt-mark hold kubelet kubeadm kubectl

#step 4:
kubeadm version
kubectl version --client
kubelet --version

#step 5:
sudo kubeadm upgrade plan

#step 6:
sudo kubeadm upgrade apply <version>
#I used the following command
sudo kubeadm upgrade apply v1.31.13

#step 7:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Now lets setup the worker nodes:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

ssh to <worker-node> node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-key

Raspberry Pi K8S Cluster Setup for Home Lab with Cilium

Tue, 17 Mar 2026 11:01:15 +0000

title: Raspberry Pi K8S Cluster Setup for Home Lab with Cilium
published: true
date: 2025-06-06 16:13:49 UTC
tags: kubernetes,raspberrypi,docker,kubernetescluster

canonical_url: https://medium.com/@mahinshanazeer/raspberry-pi-k8s-cluster-setup-for-home-lab-with-cilium-c861f7815511

I’m setting up a Kubernetes cluster using Raspberry Pis for home lab testing and learning purposes. The cluster consists of four nodes:

Master 1: Raspberry Pi 5 (8GB RAM) 192.168.1.200
Master 2: Raspberry Pi 5 (4GB RAM) 192.168.1.201
Worker 1: Raspberry Pi 3B+ 192.168.1.202
Worker 2: Raspberry Pi 3B 192.168.1.203

I’ve overclocked the Raspberry Pis to enhance performance. Below are the detailed specifications and overclocking configurations.

root@master1:/home/admin# neofetch
       _,met\(\)$gg. root@master1 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 5 Model B Rev 1.0 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-2712 
`d\)' ,$P"' . \($ Uptime: 2 hours 
 \)P d$' , \(P Packages: 1099 (dpkg), 2 (snap) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 2.700GHz 
 Y\). `.`"Y\(\)P"' Memory: 360MiB / 8063MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@master1:/home/admin#

root@master2:/home/admin# neofetch
       _,met\(\)$gg. root@master2 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 5 Model B Rev 1.0 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-2712 
`d\)' ,$P"' . \($ Uptime: 1 hour, 6 mins 
 \)P d$' , \(P Packages: 783 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 2.700GHz 
 Y\). `.`"Y\(\)P"' Memory: 234MiB / 4050MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@master2:/home/admin#  

admin@worker1:~ $ sudo su
root@worker1:/home/admin# neofetch
       _,met\(\)$gg. root@worker1 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 3 Model B Plus Rev 1.3 
',\)P ,ggs. `\(b: Kernel: 6.6.74+rpt-rpi-v8 
`d\)' ,$P"' . \($ Uptime: 1 min 
 \)P d$' , \(P Packages: 751 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' Resolution: 1024x768 
 Y\). `.`"Y\(\)P"' CPU: (4) @ 1.400GHz 
 `\(b "-.__ Memory: 166MiB / 907MiB 
  `Y\)
   `Y\(.                                              
     `\)b.                                            
       `Y$$b.  
          `"Y$b._
              `"""

root@worker1:/home/admin# 

root@worker2:/home/admin# neofetch
       _,met\(\)$gg. root@worker2 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 3 Model B Rev 1.2 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-v8 
`d\)' ,$P"' . \($ Uptime: 1 hour, 6 mins 
 \)P d$' , \(P Packages: 743 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 1.400GHz 
 Y\). `.`"Y\(\)P"' Memory: 116MiB / 906MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@worker2:/home/admin#te

First, we’ll configure m aster2 (Raspberry Pi 5–4GB) as the initial control plane node , while master1 , worker1 , and worker2 will function as worker nodes.

This setup is intentional — it allows us to explore advanced scenarios in future blog posts, such as:

Setting up a new cluster using kubeadm with Cilium.
Handling control plane degradation and recovery

Starting simple provides a solid foundation while leaving room to scale and experiment.

Preparation in each node:

The following commands prepare the OS for installing external components like container runtimes (Docker/containerd) and Kubernetes tools (kubeadm, kubelet, kubectl).

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl apt-transport-https ca-certificates software-properties-common

Enabling cgroup for CPU and memory

Hence, we are using Raspberry Pi, we can simply edit the cmdline.txt file in the firmware location to enable the cGroups.

sudo nano /boot/firmware/cmdline.txt

#add the following content
ipv6.disable=1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Now disable swap in all nodes.

Disabling swap on Kubernetes nodes is a common practice due to its potential to negatively impact performance and predictability, especially for containerized workloads. Swap, while it allows a system to use more memory than available RAM, accesses data from the disk, which is significantly slower than RAM. This can lead to performance degradation, increased latency, and unpredictable behaviour, particularly under load.

Disabling swap is mandatory because Kubernetes requires it for stable and predictable performance:

Kubernetes relies on the kubelet to manage resources, assuming memory is fully available.
Swap can cause nodes to become slow or unresponsive as critical processes get swapped out.
Disabling swap ensures the scheduler and kubelet have accurate memory availability data.
Kubernetes officially fails to start if swap is enabled unless explicitly configured otherwise (which is not recommended).

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo systemctl disable dphys-swapfile

#or

sudo swapoff -a
sudo apt purge -y dphys-swapfile

#verify using 
free -mh

Enable required kernel modules:

Enabling overlay and br_netfilter kernel modules is essential for Kubernetes networking:

overlay: Supports overlay network drivers like Calico or Flannel, enabling pods across different nodes to communicate through virtual networks.

br_netfilter: Allows Linux bridge firewall rules to inspect and filter bridged network traffic, which is crucial for Kubernetes network policies and packet forwarding.

sudo modprobe overlay
sudo modprobe br_netfilter

Adding overlay and br_netfilter to /etc/modules-load.d/k8s.conf

Adding overlay and br_netfilter to /etc/modules-load.d/k8s.conf ensures these kernel modules:

Load automatically at boot time
Maintain persistent availability without manual intervention after reboot

This guarantees Kubernetes networking components relying on these modules are always operational, avoiding network failures after node restarts.

vi /etc/modules-load.d/k8s.conf

overlay
br_netfilter

k8s.conf

Configure sysctl:

The sysctl settings in /etc/sysctl.d/k8s.conf enable essential kernel networking features for Kubernetes:

net.bridge.bridge-nf-call-ip6tables = 1

Ensures IPv6 bridged network traffic is passed through ip6tables for filtering and firewall rules.

net.bridge.bridge-nf-call-iptables = 1

Ensures IPv4 bridged network traffic is processed by iptables, allowing Kubernetes to enforce network policies and firewall rules on pod traffic.

_net.ipv4.ip_forward = 1_

Enables IP forwarding at the kernel level, allowing packets to be routed between different network interfaces — critical for pod-to-pod communication across nodes.

These settings are required for Kubernetes networking to function properly and to enforce network security policies.

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl - system

sysctl.d/k8s.confg

2. Hostname and IP configuration in the router.

To ensure network stability within the Kubernetes cluster, it’s essential to assign static IP addresses to all nodes. In my setup, I have four Raspberry Pi devices connected to my home router via Wi-Fi.

While it’s widely recommended to use wired (RJ45) Ethernet connections for Kubernetes clusters, mainly for improved reliability and lower latency, I’ve opted to use Wi-Fi. Despite the general concerns, I’ve tested this configuration extensively under various conditions and, so far, have encountered no issues.

To maintain consistent IP addressing, I reserved the IPs for each Raspberry Pi in the router’s DHCP settings. This ensures the cluster nodes remain discoverable and reachable, even after reboots or network interruptions.

I am using a Netlink Router at home. Please find my current configuration in the following screenshot. If you are using similar models, you can refer to the following configuration:

Once this is configured, we have to edit the host file in each node and add the following content:

vi /etc/hosts

#add the following
192.168.1.200 master1
192.168.1.201 master2
192.168.1.202 worker1
192.168.1.203 worker2

3. Installing Container Runtime.

In Kubernetes, the smallest deployable unit is the Pod, which usually runs one or more containers. To run these containers, Kubernetes relies on a container runtime.

Initially, Kubernetes was designed to work closely with Docker, which was the most widely adopted container engine at the time. However, over time, Kubernetes evolved, and the need to support multiple container runtimes became clear.

To address this, the Open Container Initiative (OCI) was introduced — a set of industry standards that define how containers should be built and executed. In line with OCI, Kubernetes implemented the Container Runtime Interface (CRI), which allows it to interact with any OCI-compliant runtime, such as containerd or CRI-O, in a standardised way.

Today, Docker is no longer a single, unified project. It consists of several components, including Docker-CLI, Docker-Compose, Docker Network, runc, and containerd. While containerd complies with the OCI standard and is capable of running containers, it does not natively support the CRI that Kubernetes requires.

To bridge this gap, Kubernetes initially provided a compatibility layer called Docker shim, allowing it to communicate with the Docker Engine. However, Dockershim has now been deprecated, and its role is continued by the community-maintained project CRI-Dockerd.

In this project, we are using the Docker Engine as the container runtime for Kubernetes, integrated via CRI-Dockerd.

Let’s move on and see how to install Docker and configure it as a container runtime for Kubernetes using CRI-Dockerd. Currently, Mirantis is managing the CRI-Dockerd project, and I followed their official documentation for configuring CRI-Dockerd on my Raspberry Pi. Please note that we have to configure the container runtime in all nodes for setting up the cluster successfully.

Installing Docker on all nodes

Debian

We’ll begin by installing Docker on all machines in the cluster. If you’re using a Debian-based system (such as Ubuntu or Raspberry Pi OS), you can use the following commands regardless of the system architecture:

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Once Docker installation is completed, run the following command to verify the installation is proper and working fine.

docker run hello-world

The following is the expected output;

docker run hello-world

Installing CRI-Dockerd on all nodes

You can refer to the following documentation for more details regarding CRI-Dockerd configuration:

Install

In this setup, I’m using Raspberry Pi devices, which are based on the ARM64 architecture. Instead of using the default package manager, I’m downloading the Docker binaries directly from the official release page to ensure compatibility and control over the installation version.

Releases · Mirantis/cri-dockerd

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.17/cri-dockerd-0.3.17.arm64.tgz

#after downloading extract and move it to local binary

tar -xvf cri-dockerd-0.3.17.arm64.tgz
cd cri-dockerd
sudo mv cri-dockerd /usr/local/bin/

Once the installation is completed, run the following command to verify the installation;

sudo systemctl status cri-docker.service

If the configuration is correct, running the systemctl command should return an output indicating that the service is active and running.

node1

node2 (master)

node 3

node 4

3. Installing Kubeadm

Installing kubeadm

In this setup, we will be using Kubeadm version 1.29. This specific version is chosen intentionally, as we are planning a follow-up blog focused on Kubeadm upgrade procedures. Starting with v1.29 provides a practical baseline for demonstrating version upgrades.

If you prefer to install the latest available version, you can refer to the official Kubernetes documentation linked above. Otherwise, to proceed with Kubeadm v1.29, follow the version-specific installation steps documented here.

Installing kubeadm

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

#if you are facing any issues with the installation and you want to reinstall the packages, please unhold them first.

sudo apt-mark unhold kubelet kubeadm kubectl

kubeadm, kubelet and kubectl installation completed

Follow the installation process on all nodes. Once the installation is completed, enable the kubelet on all nodes:

sudo systemctl enable --now kubelet

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.

4. Cgroup Driver Configuration

Why Cgroup Driver Configuration Matters in Kubeadm Installation?

During a Kubernetes cluster setup with Kubeadm, it is essential to ensure that the cgroup driver used by the container runtime (e.g., Docker, containerd) matches the driver expected by Kubelet, the Kubernetes node agent.

Cgroups (control groups) are used by the Linux kernel to limit and isolate resource usage (CPU, memory, etc.) of process groups. Kubernetes relies on this mechanism to enforce resource limits on containers.

There are two common cgroup drivers:

cgroupfs — a native driver provided by the Linux kernel
systemd — integrates with the systemd init system for resource management

Why this matters:

If the cgroup drivers between Docker/containerd and Kubelet are mismatched, Kubelet will fail to start, or you may encounter node stability issues.

Best practice:

Kubernetes now recommends using the systemd cgroup driver for better compatibility and integration with modern Linux distributions.

You can verify and configure the cgroup driver to ensure consistency before initialising the cluster with kubeadm. Refer to the following documentation for more details. In our case, it is not required.

Configuring a cgroup driver

5. Initialising master-node.

Creating a cluster with kubeadm

When you run kubeadm init, Kubernetes performs several critical actions to bootstrap the control plane:

Generates PKI Certificates
Secure communication between components is enabled by generating TLS certificates under /etc/kubernetes/pki.
Bootstrap the Kubeconfig Files.
Configuration files required by kubectl and system components are created under /etc/kubernetes/ and $HOME/.kube/config.
Start Control Plane Components
These pods run on the master node and are defined under /etc/kubernetes/manifests/. The following static pods are created and managed by the kubelet
- kube-apiserver
- kube-controller-manager
- kube-scheduler
Initialises the etcd Database
An embedded or external etcd key-value store is initialised to persist cluster state.
Sets up the Bootstrap Token and Join Configuration
A token and join command are generated to allow worker nodes to securely join the cluster.
Outputs Post-Initialisation Instructions
The command displays steps to:
- Configure kubectl access
- Install a Pod network add-on (required before scheduling workloads)
- Join worker nodes using the kubeadm join command

Now run the following command. After initialising the control plane with kubeadm, you need to configure kubectl to communicate with the cluster. The following commands handle that setup:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

The export command sets the environment variable so that kubectl uses the admin configuration directly from the system path. Now run the following command to initialize the master node;

sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock

If the control plane initialisation is successful, you’ll see a message similar to the one shown in the screenshot below. This message confirms that the Kubernetes control plane is up and running.

Additionally, a bootstrap token will be generated. This token, along with the control plane IP and certificate hash, is required for joining worker nodes to the cluster.

You’ll also receive instructions to:

Set up your kubectl config
Install a pod network add-on
Use the kubeadm join command on worker nodes

kubeadm init

Note: You must use the corresponding sockets depending on the container runtime version you are usi