Deploy Rook Ceph Storage on a Kubernetes Cluster

Ceph inside Kubernetes only works if you hand it storage carrying no filesystem, whether that is a whole disk, an unformatted partition or a bare LVM volume. That one requirement is where most Rook deployments stall, long before anything reaches a dashboard, because a device that already holds a filesystem is invisible to the operator and it will not tell you why.

Original content from computingforgeeks.com - post 105700

Rook is a Kubernetes operator that installs, configures and babysits Ceph for you. You describe the cluster you want in a CephCluster resource, point it at the disks, and the operator creates the monitors, managers and OSDs as pods, then wires up the CSI drivers so PersistentVolumeClaims turn into real Ceph volumes. This guide covers the whole path on a four node kubeadm cluster: preparing the raw devices, installing the operator, creating the Ceph cluster, adding a block storage class and a shared CephFS storage class, opening the Ceph dashboard, and killing a node to see what actually happens to your data. Every command, output and timing below came off a lab built in August 2026 running Kubernetes 1.36.3, containerd 2.3.3 and Ubuntu 24.04.4, with Rook v1.20.4 deploying Ceph 20.2.2 Tentacle.

What Rook adds to a Kubernetes cluster

Three moving parts, and it helps to keep them separate in your head.

The Rook operator is a single deployment that watches custom resources and reconciles Ceph daemons into existence. The Ceph cluster is the monitors, managers, OSDs and optional metadata servers, all running as ordinary pods on your nodes, storing data on the disks you gave them. The Ceph CSI drivers are what Kubernetes talks to when a pod asks for storage. In current Rook releases those drivers are managed by a separate Ceph CSI operator, which matters when you install, as you will see shortly.

What you get out of it is two storage classes with very different characters. RBD gives you fast single writer block volumes, which is what a database wants. CephFS gives you ReadWriteMany volumes that several pods on different nodes can mount at once, which is what a shared media directory or a web upload folder wants. Compare that with Longhorn, which is simpler to run but block only, or with local path provisioners, which pin a volume to one node forever.

Cluster and disk requirements

Ceph replicates across failure domains, and the default failure domain is the host. Three storage nodes is the floor, not a suggestion: with two, a replicated pool of size 3 can never go clean, and monitor quorum has no majority to fall back on when one node reboots.

The lab here is one control plane node and three workers, each worker carrying a second unpartitioned 30 GB disk:

Node	vCPU	RAM	OS disk	Ceph disk
k8s-cp	2	4 GB	40 GB	none
k8s-worker-1	2	6 GB	40 GB	30 GB raw
k8s-worker-2	2	6 GB	40 GB	30 GB raw
k8s-worker-3	2	6 GB	40 GB	30 GB raw

Those are lab numbers and they are a floor. What drives real sizing is the OSD count and the working set: budget roughly 4 GB of RAM per OSD (that is the default osd_memory_target), plus about 1 GB for a monitor. An active CephFS metadata server defaults to a 4 GiB cache under mds_cache_memory_limit, so account for that too if you use shared volumes. Those figures are ceilings a busy daemon grows into, not memory reserved up front. Production clusters use NVMe rather than spinning disks, a separate network for replication traffic, and more than one OSD per node so recovery has somewhere to go. If you want Ceph on bare metal instead of inside Kubernetes, the cephadm route on Rocky Linux is a better fit.

You also need a working cluster before Rook is any use. Anything current will do, built with kubeadm on Ubuntu or otherwise, as long as it is on a Kubernetes version Rook supports. Rook v1.20 covers Kubernetes v1.31 through v1.36 per the upstream prerequisites. Install lvm2 on every storage node as well. Raw devices like the ones used here do not strictly require it, but ceph-volume falls back to LVM as soon as you add encryption, a separate metadata device, or more than one OSD per disk, and in those cases a node without the package will create its OSDs happily and then fail to start them again after a reboot. A running udev daemon is required either way, because Rook mounts /run/udev from the host into the OSD pods and ceph-volume identifies devices from what it finds there.

sudo apt install -y lvm2
sudo modprobe rbd

The rbd module is what lets a node map a Ceph block device. Ubuntu ships it as a loadable module rather than compiling it in, so modprobe returning silently is the answer you want. Trimmed kernels may not carry it at all, and Rook singles out Google’s Container-Optimized OS as an image that ships without RBD entirely.

Check the disks Rook will consume

Do this before touching any manifest. Run it on every storage node:

lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT /dev/vdb
blkid /dev/vdb

An empty FSTYPE column and no output at all from blkid is what the operator needs to see:

NAME SIZE FSTYPE MOUNTPOINT
vdb   30G

If blkid prints a UUID or a type, that device is already claimed and Rook will skip it in silence. The OSD prepare job logs a line about the device being unavailable and moves on, leaving you with a cluster that reports zero OSDs and no obvious reason. Wipe it first, and be certain of the device name before you do, because this is not reversible:

sudo wipefs -a /dev/vdb
sudo sgdisk --zap-all /dev/vdb
sudo dd if=/dev/zero of=/dev/vdb bs=1M count=100 oflag=direct,dsync

Wiping the disks is only half of a teardown. When you delete a Rook cluster and want to reinstall on the same hardware, the dataDirHostPath directory has to go as well, and it has to go on every node rather than only the ones carrying OSDs, because monitors keep their state there too:

sudo rm -rf /var/lib/rook

Then wipe each OSD disk with the three commands above. If a disk previously held an LVM mode OSD, clear the leftover /dev/ceph-* and /dev/mapper/ceph--* entries too, since zeroing 100 MiB at the head of the device leaves the LVM metadata copies written further in. Skipping any of this is the single most common reason a second install never produces an OSD.

Install the Rook operator

Clone the repository at a release tag rather than master. The tag pins both containers you are about to run: operator.yaml carries the Rook operator image and cluster.yaml carries the Ceph image, so a tagged checkout keeps the manifests and the images in step. Rather than typing a tag that goes stale, ask GitHub which one is current:

export ROOK_VERSION=$(curl -sL https://api.github.com/repos/rook/rook/releases/latest | grep tag_name | head -1 | sed 's/.*"\(v[^"]*\)".*/\1/')
echo "${ROOK_VERSION}"

The echo prints the tag the clone is about to check out:

v1.20.4

Read that line before continuing. GitHub returns the release published most recently, which is not always the highest version number, and Rook backports fixes to older minor branches on the same day it ships current ones. If the tag is on an older branch than you want, set ROOK_VERSION by hand from the release list, which shows every branch side by side. Then clone it:

git clone --single-branch --branch "${ROOK_VERSION}" --depth 1 https://github.com/rook/rook.git
cd rook/deploy/examples

Four files, in this order, and the third one is the one older guides do not mention:

kubectl create -f crds.yaml -f common.yaml -f csi-operator.yaml
kubectl create -f operator.yaml

Leave csi-operator.yaml out and the install half succeeds, which is worse than failing. The operator Deployment at the end of operator.yaml is created and starts normally, while the three CSI resources defined above it are rejected:

resource mapping not found for name: "ceph-csi-operator-config" namespace: "rook-ceph" from "operator.yaml": no matches for kind "OperatorConfig" in version "csi.ceph.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "rook-ceph.rbd.csi.ceph.com" namespace: "rook-ceph" from "operator.yaml": no matches for kind "Driver" in version "csi.ceph.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "rook-ceph.cephfs.csi.ceph.com" namespace: "rook-ceph" from "operator.yaml": no matches for kind "Driver" in version "csi.ceph.io/v1"
ensure CRDs are installed first

Ceph itself will then come up perfectly healthy and every PersistentVolumeClaim you create will sit in Pending forever, because nothing is listening on the provisioner name your storage class points at. The Driver and OperatorConfig kinds live in csi-operator.yaml, which is roughly 390 KB of CRDs and RBAC on its own.

Wait for the operator to report ready, then look at what came up alongside it. This second command is the checkpoint that catches a missing csi-operator.yaml in minutes rather than an hour later. Give it half a minute first, because the CSI operator creates those workloads itself and they land roughly twenty seconds behind its own rollout rather than with it.

kubectl -n rook-ceph rollout status deploy/rook-ceph-operator
kubectl -n rook-ceph get ds,pods

The names are the confusing part, because the CSI operator builds them from the provisioner rather than from Rook’s own deployment names. You want a rook-ceph.rbd.csi.ceph.com-nodeplugin and a rook-ceph.cephfs.csi.ceph.com-nodeplugin DaemonSet, a matching -ctrlplugin Deployment for each driver at two replicas, and the ceph-csi-controller-manager pod. Both nodeplugins ship without tolerations, so they skip a tainted control plane: on this four node cluster they read three desired where calico-node and kube-proxy read four. Nothing here waits on a CephCluster, which is exactly why the check is worth running before you create one.

Create the Ceph cluster

The shipped cluster.yaml is 19 KB of commented options and it defaults to useAllDevices: true. On a lab VM that is harmless. On a node with a spare disk you were saving for something else, it is not, because Rook takes every unused block device it can find. Write your own instead:

vim ceph-cluster.yaml

This is the whole manifest. Twenty seven lines replaces the example file:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v20.2.2
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
    modules:
      - name: rook
        enabled: true
  dashboard:
    enabled: true
    ssl: true
  crashCollector:
    disable: false
  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^vdb$"

The deviceFilter is a regular expression matched against device names, so anchoring both ends keeps it from grabbing vdb1 or vdbx. Nodes without a matching device are simply passed over, which is one of two reasons the control plane needs no exclusion rule here. The other is its NoSchedule taint, which keeps OSDs off it regardless of what disks it has. Where node names differ or only some machines should carry storage, drop useAllNodes to false and list nodes and devices explicitly instead.

One field deserves more attention than it usually gets. dataDirHostPath is a real directory on every node holding monitor databases and daemon configuration, and it survives the cluster being deleted. That is deliberate, and it is also why a reinstall on dirty nodes behaves strangely.

kubectl create -f ceph-cluster.yaml

Watch the cluster come up

The operator pulls the Ceph image onto every storage node, roughly 550 MB over the wire that unpacks to about 1.5 GB on disk, then creates monitors one at a time, waiting for quorum between each. On this lab it took six minutes from kubectl create to a ready cluster. Watch the phase rather than the pod list:

kubectl -n rook-ceph get cephcluster -w

Progressing flips to Ready and the health column fills in:

NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          32m   Ready   Cluster created successfully   HEALTH_OK              f0f379c3-f07c-403c-94ff-0f2d3aea5f3e

To talk to Ceph directly you need the toolbox pod, which carries the Ceph CLI with the admin keyring already mounted:

kubectl create -f toolbox.yaml
kubectl -n rook-ceph rollout status deploy/rook-ceph-tools

Every ceph command below runs inside that pod. Saving the name in a variable keeps the lines short:

TOOLS=$(kubectl -n rook-ceph get pod -l app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}')
kubectl -n rook-ceph exec $TOOLS -- ceph status

Three monitors in quorum, two managers with one active, three OSDs up and in:

  cluster:
    id:     f0f379c3-f07c-403c-94ff-0f2d3aea5f3e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 4m) [leader: a]
    mgr: b(active, since 12m), standbys: a
    osd: 3 osds: 3 up (since 4m), 3 in (since 26m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   81 MiB used, 90 GiB / 90 GiB avail
    pgs:     1 active+clean

The CRUSH map is worth a look too, because it is the thing that decides where replicas land:

kubectl -n rook-ceph exec $TOOLS -- ceph osd tree

Each OSD sits under its own host bucket, which is exactly what makes host level failure domains work:

ID  CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
-1         0.08789  root default
-5         0.02930      host k8s-worker-1
 1    hdd  0.02930          osd.1              up   1.00000  1.00000
-3         0.02930      host k8s-worker-2
 2    hdd  0.02930          osd.2              up   1.00000  1.00000
-7         0.02930      host k8s-worker-3
 0    hdd  0.02930          osd.0              up   1.00000  1.00000

Note the hdd device class on virtual disks. Ceph guesses from the rotational flag the hypervisor exposes, and it guesses wrong on VM disks backed by SSDs. It only matters if you later write CRUSH rules that select by device class.

Create a block storage class

A pool and a storage class ship together in one example file, and it needs no edits for a three node cluster:

kubectl create -f csi/rbd/storageclass.yaml
kubectl patch storageclass rook-ceph-block -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

The pool is created with failureDomain: host and size: 3, meaning three copies of every object on three different nodes. Confirm the operator finished building it:

kubectl -n rook-ceph get cephblockpool

Phase Ready means Ceph accepted it:

NAME          PHASE   TYPE         FAILUREDOMAIN   AGE
replicapool   Ready   Replicated   host            22m

Now claim a volume and attach it to something:

vim block-test.yaml

A 5 GiB claim and a pod that does nothing but hold it open:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: block-writer
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: ceph-block-pvc

Apply both objects together:

kubectl apply -f block-test.yaml

Binding took under twenty seconds here. Look at what actually landed in the pod:

kubectl exec block-writer -- df -hT /data

An ext4 filesystem on a kernel RBD device, not a network mount:

Filesystem           Type            Size      Used Available Use% Mounted on
/dev/rbd0            ext4            4.8G     24.0K      4.8G   0% /data

Write half a gigabyte through it with a durable flush at the end:

kubectl exec block-writer -- dd if=/dev/zero of=/data/test.bin bs=1M count=512 conv=fsync

66.8 MB/s, on virtual disks sharing one host’s storage, with every block written three times:

512+0 records in
512+0 records out
536870912 bytes (512.0MB) copied, 7.660313 seconds, 66.8MB/s

That number is the honest cost of replication on modest hardware, and it is the one people are surprised by. Reading it back afterwards returned 421.5 MB/s, mostly out of cache. On real NVMe with a dedicated cluster network you land in a different league entirely, but the shape holds: writes pay for durability, reads do not.

The claim shows up in Ceph as an RBD image in the pool:

kubectl -n rook-ceph exec $TOOLS -- rbd ls -p replicapool

The name is derived from the volume handle rather than from the claim, so it will not look like anything you typed. The dashboard lists the same image with its size and enabled features filled in:

Delete the pod and recreate it and the data is still there, which is the entire point. If you are coming from an external Ceph cluster wired up by hand, this is the part that manual ceph-csi setups make you assemble yourself.

Add CephFS for shared volumes

RBD volumes are single writer. When three replicas of a deployment need the same directory, you need a filesystem, which means a metadata server:

kubectl create -f filesystem.yaml
kubectl create -f csi/cephfs/storageclass.yaml

That creates a metadata pool, a data pool, and two MDS daemons in an active plus hot standby pair. Give it a minute, then check from the Ceph side:

kubectl -n rook-ceph exec $TOOLS -- ceph fs status

standby-replay is the important word: the standby is already tailing the active MDS journal, so failover is fast rather than a cold start:

myfs - 1 clients
====
RANK      STATE        MDS       ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active      myfs-a  Reqs:    1 /s    12     15     14      3
0-s   standby-replay  myfs-b  Evts:    1 /s     2      5      4      0
      POOL         TYPE     USED  AVAIL
 myfs-metadata   metadata   160k  27.9G
myfs-replicated    data       0   27.9G

Two storage classes are live now:

kubectl get sc

Block on the left, shared file on the right, both expandable:

NAME                        PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block (default)   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   22m
rook-cephfs                 rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   18m

Prove the shared part properly, with pods forced onto separate nodes:

vim shared-test.yaml

A ReadWriteMany claim and a three replica deployment with a topology spread constraint, each replica writing a file named after itself:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-shared
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: rook-cephfs
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shared-writer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shared-writer
  template:
    metadata:
      labels:
        app: shared-writer
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: shared-writer
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "echo written by $HOSTNAME > /shared/$HOSTNAME.txt; sleep 3600"]
          volumeMounts:
            - name: shared
              mountPath: /shared
      volumes:
        - name: shared
          persistentVolumeClaim:
            claimName: cephfs-shared

Roll it out and wait for all three to land:

kubectl apply -f shared-test.yaml
kubectl rollout status deploy/shared-writer

List the directory from any one of the three pods:

kubectl exec deploy/shared-writer -- ls -l /shared

All three files, written from three different nodes, visible from every replica:

total 2
-rw-r--r--    1 root     root            42 Aug 14 01:55 shared-writer-65d9bff757-4plfz.txt
-rw-r--r--    1 root     root            42 Aug 14 01:55 shared-writer-65d9bff757-jr86t.txt
-rw-r--r--    1 root     root            42 Aug 14 01:55 shared-writer-65d9bff757-kdhvw.txt

One behaviour to expect on a fresh write: for a second or two, a file created on one node can show a size of 0 to a client on another node while its contents already read back correctly. CephFS clients hold capabilities on a file and the size only becomes authoritative once the writer flushes to the metadata server. Applications that stat a file the instant it appears on a peer node need to account for that.

The quota is real, incidentally. df inside the pod reports 2.0G, not the cluster’s free space, because the CSI driver sets a directory quota on the subvolume.

Four pools now, all at three times replication. Which brings up the arithmetic nobody enjoys:

kubectl -n rook-ceph exec $TOOLS -- ceph df

90 GiB of raw disk, and 28 GiB usable, shown against every pool but drawn from one shared free space:

--- RAW STORAGE ---
CLASS    SIZE   AVAIL     USED  RAW USED  %RAW USED
hdd    90 GiB  88 GiB  1.6 GiB   1.6 GiB       1.81
TOTAL  90 GiB  88 GiB  1.6 GiB   1.6 GiB       1.81

--- POOLS ---
POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr              1    1  449 KiB        2  1.3 MiB      0     28 GiB
replicapool       2   32  515 MiB      148  1.5 GiB   1.77     28 GiB
myfs-metadata     3   16  264 KiB       25  889 KiB      0     28 GiB
myfs-replicated   4   32    925 B        6   72 KiB      0     28 GiB

Divide by three for replication, then subtract the full ratio headroom Ceph reserves, and 90 GiB becomes 28. That number is an estimate, not an allocation. Every pool reports the same 28 GiB because every pool is drawing from the same free space, so filling one empties the others. Size the disks accordingly, because a Ceph cluster that hits the full ratio stops accepting writes rather than degrading gracefully.

Open the Ceph dashboard

The manager runs a dashboard and Rook enables it by default, generating an admin password into a secret:

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d; echo

The service is ClusterIP only, so forward a local port to reach it:

kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443

Browse to https://localhost:8443 and log in as admin. The certificate is self signed, so expect a browser warning. For anything longer lived than a debugging session, put an Ingress in front of it with a real certificate rather than leaving a port forward running.

Orchestrator reads rook, which is how the dashboard knows it is not driving a cephadm cluster and hides the deployment controls that would not work here. The inventory counts four hosts even though only three carry OSDs, because the rook module builds that list from Kubernetes Node objects rather than from Ceph daemons.

Two panels will look broken and are not. The Cluster Utilization graphs stay empty and the dashboard throws a 404 - Not Found, Could not reach Prometheus's API toast, because those charts are rendered from Prometheus and nothing has told the dashboard where one lives. Point it at an existing instance with ceph dashboard set-prometheus-api-host, or install Prometheus and Grafana first and set it afterwards. Everything else on the page comes straight from the manager and is live.

From here the same cluster will happily serve S3 compatible object storage through a CephObjectStore, and the toolbox pod is where you will spend most of your time when something needs investigating.

What happens when a storage node dies

Replication is a claim until you test it. So: hard power off on k8s-worker-3, which was running a monitor, an OSD and an MDS, while both test volumes stayed mounted. Timings measured from the moment the VM stopped.

Elapsed	What Ceph did
10 s	Monitor marked down, quorum held on the surviving two
28 s	OSD marked down, 178 of 534 objects now degraded
63 s	Filesystem flagged degraded, standby MDS promoted
112 s	Filesystem healthy again on the promoted MDS
7 min	OSD still marked in, no rebalancing started

All 81 placement groups stayed active for the entire outage. Degraded, undersized, but active:

81 pgs: 30 active+undersized, 51 active+undersized+degraded; 522 MiB data, 1.1 GiB used, 59 GiB / 60 GiB avail; 178/534 objects degraded (33.333%)

Both volumes kept serving reads and writes throughout, because the pool’s min_size is 2 and two replicas were still up. Lose a second node out of three and that stops: PGs go inactive and every pod blocks on I/O until a node returns. Three nodes tolerates one failure, and only one.

The last row of that table is the one worth internalising. Ceph waits mon_osd_down_out_interval seconds, 600 by default, before marking a down OSD out and re-replicating its data elsewhere. That grace period is deliberate, since most node outages are reboots and rebalancing 30 GB across a cluster to undo it five minutes later helps nobody. On a three node cluster with a host failure domain it would have been futile anyway, as there is no third host left to hold the third copy.

Kubernetes had its own opinion during all this. The RBD volume’s pod was on a surviving node and never noticed. The CephFS deployment lost one replica and could not reschedule it, since a topology spread constraint of DoNotSchedule across hostnames had nowhere left to put it:

0/4 nodes are available: 2 node(s) didn't match pod topology spread constraints, 2 node(s) had untolerated taint(s).

Powering the node back on: 81 seconds from boot to HEALTH_OK, monitor rejoining quorum, OSD coming back up and backfilling the 179 objects it had missed. No manual step anywhere.

Worth knowing what that costs to run. With Ceph healthy and both volumes in use, the storage nodes were sitting between 0.9 GB and 1.5 GB of RAM each, covering the monitor, the OSD, whichever manager or MDS landed there, the CSI plugins, Calico and the kubelet. Nowhere near the 4 GB per OSD from the sizing table, because osd_memory_target is a ceiling a busy OSD grows into, not memory claimed at startup. Rook and Ceph are frequently described as heavy, and they are demanding on disks and on your attention. Memory, at this scale, is not where the weight is.