Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide

Sat, 21 Mar 2026 13:53:24 +0000

title: Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide
published: true
date: 2025-10-16 14:35:59 UTC
tags: raspberrypi,kubernetes,etcd,kubernetescluster

canonical_url: https://medium.com/@mahinshanazeer/upgrading-our-raspberry-pi-kubernetes-cluster-from-v1-29-to-v1-34-etcd-backup-guide-97a2074237b4

Keeping your cluster up-to-date is crucial for security, stability, and access to new features. In this blog, we’ll take a closer look at our home lab Kubernetes cluster running on Raspberry Pi devices. The setup includes 1 master node and 3 worker nodes, and it has been running smoothly for over four months without any major issues.

k8s home lab configuration — nodes

master2 — master node/ control plane
master1 — worker node 3
worker 1 — worker node 1
worker 2 — worker node 2

k8s home lab configuration — system components

As part of regular maintenance and to stay up-to-date with the latest features and security patches, we decided to upgrade the cluster from Kubernetes v1.29 to v1.34. Alongside the upgrade, we’ll also walk through the process of backing up the ETCD datastore, which is crucial for fault tolerance and disaster recovery.

Here’s what we’ll cover:

⬆️ Step-by-step guide to upgrading Kubernetes components.
️ Safely backing up ETCD in the master node.
✅ Post-upgrade verification.

A. Backing Up ETCD

Before diving into the upgrade process, it’s essential to create a backup of ETCD, the key-value store that holds all cluster state data. This step is crucial because if anything goes wrong during the upgrade — whether it’s a misconfiguration or a failed component — we need a reliable way to restore the cluster to its current stable state.

Think of the ETCD backup as your safety net. It ensures that even in the worst-case scenario, you won’t lose your cluster’s configuration, workloads, or networking setup.

Run the following command to confirm the ETCD pod name running on your control plane:

kubectl get pods -n kube-system | grep etcd

k8s configuration — ETCD setup

Suppose your control plane (master node) runs ETCD as a static pod (which is the default in kubeadm-based clusters, including Raspberry Pi setups). In that case, you can run etcdctl directly on the master node host — because the ETCD data directory and certificates are mounted locally at /etc/kubernetes/pki/etcd/ and /var/lib/etcd/. In that case, you can use the following command to take the backup:

sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

If ETCD runs as a container (inside the pod) or you don’t have the etcdctl binary available on the host, then you’ll need to exec into the pod to run the backup from inside the container.

In my setup, where etcd runs as a container, I need to access the pod directly to execute the backup. There is no etcdctl binary available on the host.

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

#edit the <node-label> with etcd node

Creating backup

Now, verify the snapshot using the following command:

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot status /var/lib/etcd/backup.db -w table

In the above screenshot, you can seee a deprecation warning. ETCD v3.6+ is deprecated etcdctl snapshot status and recommends using etcdutl snapshot status. However, in my kubeadm static pod setup, the etcdutl binary is not included in the ETCD container. Only etcdctl exists inside the pod. The warning is simply a deprecation notice, not a failure. You can safely continue using it etcdctl snapshot status — it works perfectly.

In my setup, I came across a deprecation warning while using etcdctl. To verify, I checked the current version of etcd using the following command.

Since my cluster is kubeadm-managed, even though etcd v3.6+ recommends using etcdutl for snapshot management, etcdctl inside the etcd pod still works perfectly for backup and restore. etcdutl is a newer utility introduced in etcd v3.6+ as a replacement for certain snapshot operations, but in my case, continuing with etcdctl is fully sufficient.

kubectl exec -n kube-system etcd-master2 -- etcdctl version

etcd version checking

In some scenarios, the etcd backup file may reside only inside the etcd pod and not in a host-accessible directory.

This can happen if the etcd container is minimal and lacks tools like tar to move files around, or if the host does not have etcdctl installed to create the snapshot directly. In such cases, you can use specific commands to move the backup file to a host-accessible directory.

kubectl cp kube-system/etcd-master2:/tmp/backup.tar.gz ./backup.tar.gz

In my setup, I created the etcd snapshot directly on the host in a host-accessible directory (for example, /var/lib/etcd/backup.db). Since the backup is already on the host, it can be copied or archived without using kubectl cp.

sudo cp /var/lib/etcd/backup.db ~/backup.db
sudo chown $USER:$USER ~/backup.db
ls -l ~/backup.db

backup.db file status

Additionally, it’s a good practice to create a copy of the backup on a remote server. This ensures an extra layer of safety and helps protect against data loss in case of hardware failure or accidental deletion.

B. Upgrading the cluster

Since we have backed up etcd, we can safely proceed with the kubeadm upgrade for our Raspberry Pi Kubernetes cluster. Let’s go through a step-by-step process to upgrade the cluster from v1.29 → v1.34.

Kubernetes officially supports upgrading one minor version at a time, which means that to move from 1.29 to 1.34, you need to perform incremental upgrades:

Step 1: v1.29 → v1.30
Step 2: v1.30 → v1.31
Step 3: v1.31 → v1.32
Step 4: v1.32 → v1.33
Step 5: v1.33 → v1.34

Please refer to the following URL for k8s official documentation on skew policy:

Version Skew Policy

Step 1: Pre-Checks for upgrading the cluster

Before upgrading the cluster, it’s essential to ensure that the cluster is healthy and stable. This helps prevent issues during the upgrade and ensures a smooth transition between versions.

kubectl version
kubectl get nodes
kubectl get cs
kubectl get pods -n kube-system

Current status of the cluster

Step 2: Upgrade kubeadm on the Master Node

You can also refer to the following documentation for more details regarding kubeadm upgrade:

Upgrading kubeadm clusters

To upgrade to v1.30, ensure that the directory /etc/apt/keyrings exists. If it doesn’t, create it before running the curl command (see the note below). Afterwards, update the package repository to target v1.30.

Each Kubernetes minor release repository (v1.29, v1.30, v1.31, etc.) may use a different GPG key. Using the key from an older version, such as v1.29, can cause package verification failures when installing packages from the v1.30 repository.

To prevent such errors, always use the GPG key and repository provided specifically for the minor version you are upgrading to.

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Before upgradingkubeadmEnsure your package lists are up to date. Then, install the required version by using the following commands

sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

If you want to upgrade to a specific Kubernetes version, you can use the following command to list all available minor versions in the corresponding repository:

apt-cache madison kubeadm

available packages in the repo

For comprehensive guidance on upgrading your Kubernetes cluster to version v1.30 , refer to the official Kubernetes documentation:

Installing kubeadm

This resource provides detailed, step-by-step instructions tailored for clusters managed with kubeadm , ensuring a smooth and efficient upgrade process.

Once the upgrade is complete, verify the kubeadm version using the following command:

kubeadm version
kubectl version --client
kubelet --version

current kubeadm version

Step 3: Plan the Upgrade

After upgrading kubeadm, it’s important to check the upgrade path and verify the versions of all cluster components. This ensures that all parts of your Kubernetes cluster are compatible and running the expected versions.

sudo kubeadm upgrade plan

It will display:

The current cluster version and control plane version.
The available target versions you can upgrade to.
The versions of kubelet and kubectl on your nodes.
Any required steps or notes for a smooth upgrade.

kubeadm upgrade plan output — 1

kubeadm upgrade plan output — 2

Step 4: Upgrade the Control Plane

Once the upgrade plan is reviewed and all changes are understood, you can proceed with upgrading the cluster. Use the kubeadm upgrade apply command, which is provided in the output of kubeadm upgrade plan.

sudo kubeadm upgrade apply v1.30.14

The upgrade process may take some time, depending on the size and complexity of your cluster. Once it completes successfully, you should see a message similar to the following:

upgraded to “v1.30.14”

Now, verify the control plane using the following commands:

#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Step 5: Upgrade Worker Nodes

Now repeat the same steps from ‘Step 2: Upgrade kubeadm on the Master Node’ again in master1, worker1 and worker2 nodes (worker nodes). Workers do not control the cluster state; running upgrade plan there doesn’t provide meaningful information. First drain the worker node from the control plane, so that all the resources will be taken out of the node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

draining resources from worker node (master1)

Now you can repeat the steps we followed for control plane for upgrading the version:

ssh to master1 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
kubeadm upgrade node

upgrading worker node — host ‘master2’ is a worker node here. (master2 — master node/ control planemaster1 — worker node 3worker 1 — worker node 1worker 2 — worker node 2)

Also verify the nodes from control plane:

upgrade completed

Now we can following commmand to make the node available for schedule;

kubectl uncordon master1
#run this command in the control plane.

Marking node as schedulable

kubectl uncordon command is used to mark a node as schedulable again after it has been drained.

Similarly we can do the upgrades in other 2 worker nodes, I will mention the commands below:

worker1:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker1 node

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker1
kubectl get nodes
kubectl get pods -n kube-system

worker2:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker2 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker2 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker2

All nodes have now been successfully upgraded to v1.30.14 , as shown in the following screenshot.

kubernetes version v1.30.14

Step 5: Upgrading to 1.34

Following the same process, we’ll now proceed to upgrade the cluster from v1.30.14 to v1.31. Then we will proceed with with

Step 1: v1.29.15 → v1.30.14–1.1
Step 2: v1.30.14–1.1 → v1.31.13–1.1
Step 3: v1.31.13–1.1 → v1.32.9
Step 4: v1.32.9 → v1.33.5
Step 5: v1.33.5 → v1.34.1

In control-plane (master2) use the following steps to upgrade to version 1.31:

ssh to controlplane (master2)

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 2:
sudo apt update
apt-cache madison kubeadm

#step 3:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm kubelet kubectl #will take the latest version
sudo apt-mark hold kubelet kubeadm kubectl

#step 4:
kubeadm version
kubectl version --client
kubelet --version

#step 5:
sudo kubeadm upgrade plan

#step 6:
sudo kubeadm upgrade apply <version>
#I used the following command
sudo kubeadm upgrade apply v1.31.13

#step 7:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Now lets setup the worker nodes:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

ssh to <worker-node> node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-key

Coder Legion - Recent posts tagged etcd