Repository navigation
Cluster Audit: find unclaimed storage (unconsumed PVCs, long-Pending claims, Released PVs) - #2055
Conversation
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
PR Summary by QodoAdd storage waste checks to Cluster Audit
AI Description
Diagram
High-Level Assessment
Files changed (17)
|
Code Review by Qodo
1.
|
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
2 similar comments
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
âĶ volumes that can land on Hyperdisk-only nodes (#2069) ## Why A prod Redis StatefulSet was rescheduled onto an `n4d-standard-4` node after a node-pool change. N4D machines attach only Hyperdisk, and Redis's volumes are `premium-rwo` (pd-ssd), so the volume never attached and the pods stayed in ContainerCreating for hours. Radar reported the attach failure, but with the wrong event. `diagnose` and `issues` both showed: > `ControllerPublish not permitted on node "âĶ6uw0" due to backoff condition` That is the CSI attacher's retry notice. The event that names the cause was older and was dropped, because post-bind detection keeps only the latest event per pod: > `googleapi: Error 400: pd-ssd disk type cannot be used by n4d-standard-4 machine type., badRequest` ## What changes **Diagnose and issues (post-bind detection)** - When the latest attach event is a backoff echo, report the newest informative attach error **for the same pod UID and the same volume**. Onset is taken from the first real failure. - Parse Compute Engine's disk/machine incompatibility into a plain `Cause` and `Action`: "pd-ssd volumes can't attach to n4d-standard-4 nodes: the N4D machine series supports only Hyperdisk". The action offers two fixes: pin the workload to a compatible pool, or move it to a Hyperdisk StorageClass that every eligible pool supports. - When only echoes remain (the real event has expired), derive the same cause from the node's `node.kubernetes.io/instance-type` and the PV's provisioned disk type. - `diagnose`'s `startupBlockers` now carry `cause` and `action`. Both are optional additions. - Events recorded against a predecessor pod's UID are ignored. StatefulSet pods are recreated under the same name, so without this a dead pod's error could be shown for its replacement. **Cluster Audit: new check `pdVolumeHyperdiskOnlyNode` (Reliability)** - Flags a workload when it mounts a GCE Persistent Disk volume (`pd-*`) and its pods could be scheduled onto a node from a Hyperdisk-only machine series. - Constraints come from the **owner's current pod template** (Deployment, StatefulSet or DaemonSet). Node selector, required node affinity, volume node affinity and zone/region labels (including beta PV labels matched to stable node labels), and NoSchedule/NoExecute taints are all evaluated. Inter-pod affinity and topology spread are not, and the message says only what was checked. - Severity is **danger** when a pod is already on such a node, and **warning** when one could land there. - The disk type is trusted only for dynamically provisioned PVs (`pv.kubernetes.io/provisioned-by`). A static PV that merely borrows a class name is skipped. StorageClass parameter keys are matched case-insensitively, as the PD CSI driver matches them. - Evidence is gated per reader on list access to `nodes`, `persistentvolumes` and `storageclasses`. A reader missing any of them gets no evidence, and the check is silent. - Hyperdisk-only series come from a fixed table based on Compute Engine's disk-support matrix: A4/A4X, C4/C4A/C4D/C4N, G4, H4D, M4/M4N, N4/N4A/N4D, X4/X5, Z4D. Series with partial PD support (C3, C3D, H3) and unknown series are never flagged, so the table can only under-report. New package `pkg/gkedisk` holds the shared knowledge. `pkg/audit.CheckInput` gains a `PDPlacement` field. Both are additive for Hub. ## Decisions - **Accepted disclosure.** The cache fallback (only backoff echoes remain) reads the Node, PV and StorageClass with Radar's identity. A reader who can see the pod's namespace but can't list those kinds will see the machine type and disk type in the issue's cause. The types are low-sensitivity, and the fallback only applies once the informative event has expired. The audit check, by contrast, is gated per reader. - **Overlap with #2055.** Both PRs touch `internal/audit/scope.go`, `internal/audit/runner.go`, `pkg/audit/{types,registry,checks}.go`. This PR's edits sit in separate hunks, and `PDPlacement` is its own field rather than the PV/PVC/StorageClass inputs #2055 adds. Expect at most a mechanical rebase. ## Scenario ledger | Scenario | Expected | Proof | |---|---|---| | Real GCE error older than 10 min, latest event is a backoff echo | Message shows the real error; Cause/Action set; onset from first failure | `TestDetectPostBindProblems_AttachBackoffShowsRealCause` (fails on main) | | Only echoes remain; node n4d, PV pd-ssd | Cause derived from cache | `âĶAttachBackoffOnlyDerivesCause` (fails on main) | | Echo on a PD-capable node | No cause invented | `âĶAttachBackoffOnCompatibleNode` | | Predecessor pod UID's events, old or recent | Not used for the replacement | `âĶAttachIgnoresPredecessorPodEvents` | | Two volumes; echo for A, real error for B | A keeps its own echo | `âĶAttachEchoMatchesItsOwnVolume` | | Unrelated attach error (timeout) | No disk cause | `âĶUnrelatedAttachFailureHasNoDiskCause` | | Unpinned STS on pd-ssd, cluster has N4D | Warning on the StatefulSet | `TestPDPlacement_UnpinnedStatefulSetReachesHyperdiskOnlyNode`, `TestRunFromCacheFlagsPDVolumeReachingHyperdiskOnlyNode` | | Pod already on N4D | Danger | `âĶPodAlreadyOnHyperdiskOnlyNodeIsDanger` | | nodeSelector / NotIn affinity / untolerated taint | Evaluated, passes | `âĶConstraintsThatKeepPodsOff` | | Zonal disk, N4D only in another zone (nodeAffinity or zone labels) | No finding | `âĶVolumeZoneKeepsPodOffOtherZones`, `âĶLegacyVolumeZoneLabel` | | PV has beta zone/region labels, node has stable labels | Matching nodes remain eligible; mismatches are excluded | `âĶLegacyVolumeZoneLabel` | | Template pin added or removed vs running pod | Template wins | `âĶConstraintsComeFromOwnerTemplate` | | Mixed-case StorageClass `Type: hyperdisk-balanced` | Classified as Hyperdisk, never as the PD default | `TestDiskType` | | Static PV, Hyperdisk volume, non-GCE storage, no N4D nodes | Not evaluated | `TestDiskType`, `TestPDPlacement_NotApplicable` | | Reader can't list StorageClasses or nodes | No evidence, no finding | `TestReadScopeResolvesPDPlacementGrants`, denied-scope case in the runner test | ## Verification - `make tsc` passes. - `make test` passes. - `cd pkg && go test ./audit ./gkedisk` passes, including matching and mismatching beta-PV/stable-node zone and region cases. - `make build` passes (frontend, embed, Go binary). - Visual-test: skipped (backend diagnostics and audit logic; no rendered UI changes).
Every piece of unused storage in a real cleanup was found by hand: hundreds of GiB of disks left by deleted apps, claims Pending for years on WaitForFirstConsumer classes (which the issue detector deliberately ignores), and Released volumes stuck failing deletion. Cluster Audit now reports, as a snapshot: - Bound PVCs with no consumer Radar can observe (pods, every built-in workload template including zero-replica StatefulSets, generic ephemeral volumes), with size, class, age, PV and reclaim policy; - PVCs Pending over 24 hours, including WaitForFirstConsumer claims only when nothing references them; - Released PVs kept by a Retain policy, and those whose deletion is failing (a recent VolumeFailedDelete event, with its message). Findings say "no consumer observed", never "safe to delete". Inputs are gated on cache readiness and the caller's exact grants, so unreadable inventories report missing inputs instead of inventing absence findings.
PVCs owned by a controller (a hibernated CloudNativePG cluster's volumes, a stopped KubeVirt VM's disks, workflow volumeClaimTemplates) are skipped, since their owner manages their lifecycle; both PVC checks also wait until a claim is a day old so apply order and restores don't raise findings. Pending claims are reported only with no observed consumer, as "Pending since creation" (a claim never returns to Pending). Released Delete volumes that haven't been deleted after an hour are reported even without events; retained ones get a 24-hour grace. Missing PV or StorageClass access is reported only when a claim needed it, every readable PV counts toward the check, events are listed only when needed, and the new audit inputs have readable labels.
A recent Released volume no longer counts as passing when deletion events can't be read, a reused PV name doesn't pick up a stale deletion warning from an earlier volume (events are matched by UID), and a readable but absent StorageClass no longer reports binding mode as missing evidence.
Every consumer reads a missing input as "some checks could not run". The storage checks also listed supplemental evidence gaps there (StorageClass binding mode, PV reclaim policy, deletion warnings) for claims and volumes they still evaluated and flagged. A missing input is now recorded only where the gap left a subject unevaluated: an unreadable PVC inventory, unknown consumers, an unavailable PV inventory, or a recently Released Delete volume whose outcome depends on events Radar can't see. Gaps that still let a finding fire are stated in its message, including a new "Deletion warning events not fully visible" caveat. PVs outside a caller's grant are not subjects of that caller's scan, so they no longer report a missing input.
A PV whose in-tree deletion or recycling fails moves to phase Failed and stays there; releasedPV only judged Released volumes, so these counted as passes. Failed volumes are now always evaluated and reported, with the latest current reclaim warning or else the PV's status message (the controller records the failure there and emits the event only once). Incomplete event coverage adds the usual caveat and never skips them. Available and Pending volumes have no claim to have outlived, so they are no longer counted as releasedPV passes either.
âĶnsumer coverage
âĶction README states the Cluster Audit check count again (43, the registry size), and a test fails when the two drift. The storage review checks section in docs/configuration.md keeps its contract in half the length, and the MCP audit access bullet links to it instead of repeating it.
Main added checkPDVolumeHyperdiskOnlyNode while this branch was open, so the registry now has 44 checks and TestREADMECheckCountMatchesRegistry needs the README to say so.
7185ace to
af625a6
Compare
âĶg PVC clones as consumers
âĶsumer age floor
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
In a real cleanup every piece of unused storage was found by hand:
defaultVolumeFailedDeleteretry loopCluster Audit now reports these as snapshot Efficiency checks (Medium on the existing Checks and MCP
get_cluster_auditsurfaces):pvcNoConsumerUnusedcondition is used when present (below). Shows size, storage class, age, bound PV and reclaim policy.pvcLongPendingreleasedPVRetainpolicy (after a 24h grace when release time is known, or flagged with "release time unknown" and PV age when it is not), and those whose deletion hasn't completed after an hour of known release duration. A recentVolumeFailedDeletewarning establishes deletion failure immediately, even without a phase-transition timestamp. Delete PVs without that timestamp or a recent warning are not evaluated. Failed PVs (an in-tree delete or recycle failed after the claim was deleted) are reported immediately, with the latest currentVolumeFailedDelete/VolumeFailedRecyclewarning or else the PV'sstatus.message.Not waste by construction:
volumeClaimTemplates, follows its owner's lifecycle, so these checks leave it to the owner.Unusedtime or binding-mode evidence where present.Kubernetes 1.37
Unusedcondition (PVC protection controller, beta, on by default):Unused=False(a running Pod uses the claim) counts as a consumer, even if Radar's Pod inventory missed it.Unused=True, the 24h floor counts from when the claim became unused instead of from its creation, so a claim whose workload was deleted minutes ago isn't flagged. The finding adds "Kubernetes reports it unused by any Pod since (5d ago)". It reports the condition rather than promising a minimum idle time: upstream notes the time can be off in either direction because of controller delays. Right after the feature is enabled (e.g. an upgrade to 1.37), every unused claim's timestamp is the enable time, so long-unused claims are held back for a day and then reported with that shorter age.Unknownkeeps the creation-age behavior; absence never implies unused.pvcLongPendingdoesn't use it.Deleting claims and volumes â
stuckTerminating:stuckTerminatingsubjects, with the same 5m/30m ramp as other kinds. Deleting PVCs were already skipped by the storage checks; deleting PVs are now skipped byreleasedPVtoo, so a deleting Bound PV no longer counts as a pass and a deleting Released PV isn't reported twice.kubernetes.io/pvc-protectionandkubernetes.io/pv-protection, the finding explains what holds the deletion and what happens when it's released (the claim is deleted and the volume follows its reclaim policy), suggests a backup if the deletion was unintended, and says not to remove the finalizer. It never tells the user to stop Pods or delete the claim: a deletion can't be cancelled, and completing it can delete the data.stuckTerminatingremediation (shown in Checks and MCP) now exempts storage finalizers from its "clear finalizers as a last resort" advice.Coverage:
missingInputskeeps its existing meaning: something was not evaluated. A storage input is recorded only when its absence left a claim or volume unevaluated, so unreadable evidence never looks like a clean pass:persistentvolumeclaims)pvc-consumers)persistentvolumes)pv-deletion-events)pv-release-time)Failedonly when reclaiming failed, so the phase alone establishes the finding. Events only add detail; incomplete coverage adds the "Deletion warning events not fully visible" caveat, neverpv-deletion-events. The controller emits the warning once, at the transition, and events expire (1h by default), sostatus.messageis the fallback.releasedPVcounts: Bound PVs pass, Released and Failed PVs are judged, and Available/Pending PVs are not counted. They have no claim to have outlived, so counting them as passes would only inflate the pass rate (for example, a local-static-provisioner pool).No Radar-kept usage history, cost column or delete action; those are follow-ups.
pkg/auditchanges are additive.Testing
dataSourceRefreferences protect only the source in the referenced namespace. Bound clones, non-core references, and VolumeSnapshot references do not protect a source. Tests also cover StatefulSet replicas 0 and leftover ordinals, controller-owned (CNPG) claims, the 24h age floor at a fixed clock, WaitForFirstConsumer with and without consumers, and Retain vs Delete with and without events. Nil and zero release timestamps cover Delete skips, immediate recent-warning findings, and Retain findings with unknown release time and PV age. They also cover missing inputs only for skipped subjects (otherwise the caveat is in the finding), PV inventories that are unavailable vs. outside the caller's grant, and unreadable namespaces suppressing findings.Unusedcondition: False with no observed consumer and with incomplete inventory, True 2h ago on a 30-day-old claim, the exact 24h boundary, future and zero timestamps, True 10 days ago with the evidence sentence, zero-replica StatefulSet and Pod consumers, absent andUnknownconditions, and Pending claims ignoring it.pv-release-timeappearing only with a PV grant), and theChecksViewinput labels.status.message, with partial or unreadable events (caveat, evaluated, not passed, no missing input), with a warning older than the Failed transition (ignored), Recycle withVolumeFailedRecycle, and a policy switched to Retain after the failure. Counts: Bound passes; Failed is evaluated but not passed; Available and Pending are not counted.internal/auditcollects events for Failed PVs.make testandmake tscpass. The original change was checked live on a kind cluster (an unconsumed bound PVC and a Released Retain PV through REST and MCP).Live verification (kind, Kubernetes v1.36.1 so
lastPhaseTransitionTimeis populated; real binary in local and proxy-auth modes; REST + MCP + Checks UI)3dc5f6a3):persistentvolumesmissing input.pvc-consumers.VolumeFailedDeleteon a Released PV is reported as "deletion is failing".get_cluster_auditmatches REST.Failedwith a realVolumeFailedDeleteevent, and was counted as areleasedPVpass.4be95ee8: the binary was built from3dc5f6a3plus the uncommitted fix, whose diff is identical to the4be95ee8commit; same repro: hostPath Delete PV at/var/..., claim deleted, PVFailedwithstatus.messageset and one event indefault):releasedPVevaluated 2, passed 1 (the Bound PV passes, the Failed PV is flagged):Failed PV: reclaiming the volume failed after its claim was deleted (reclaim policy Delete). Latest reclaim warning (2m ago): host_path deleter only supports /tmp/.+ but received provided /var/fix2055-delete-fail. Capacity 1Mi; storage class ""; former claim fix2055-f3/claim-delete-fail; Failed for 2m.MCP shows the same text.defaultis hidden: the same finding, carrying theStatus message: host_path deleter only âĶtext plus "Deletion warning events not fully visible". There is nopv-deletion-eventsmissing input, and REST and MCP match.Live verification on GKE 1.37.1 (nonprod control plane upgraded to 1.37.1; current head)
Unused=Falseon a claim with a running Pod andUnused=True(timestamped at Pod deletion) on two claims whose Pods were deleted. Astandard-rwo(WaitForFirstConsumer) clone of one of them stayed Pending.orphanflagged with the Unused time (5d ago);clone-sourceprotected by the Pending clone;in-usepasses; the abandoned Pending clone flagged aspvcLongPendingin-usestill passes onUnused=Falseorphanandclone-sourceheld back (not evaluated)clone-sourceflaggedpkg/audittests; all three fail on the previous head.make test,make tscand the audit/MCP suites pass on the current head, after merging main (which brought Detect custom resources stuck deleting on an operator's finalizer #2053).Live check of stuck deletion (GKE 1.37.1, head
9ea8f6871): a PVC still mounted by a running Pod was deleted, then its bound PV. After 6 minutes both were reported bystuckTerminatingas warnings: the PVC held bykubernetes.io/pvc-protection, the PV bykubernetes.io/pv-protectionplus the two CSI finalizers, each with the protection explanation.releasedPVcounted only the cluster's other (Bound) PV, so the deleting PV was neither passed nor double-reported. Deleting the namespace then removed the PVC, PV and disk.Docs
CheckRegistry, which now includes the Persistent Disk placement check main added in Explain volume attach failures from incompatible disk types; audit PD volumes that can land on Hyperdisk-only nodes #2069);TestREADMECheckCountMatchesRegistryfails when the two drift.docs/configuration.mddescribes the consumer inventory, release-time uncertainty, thresholds, and evidence gaps; the MCP audit-access bullet links to it instead of repeating it.Rollout and compatibility
get_cluster_auditdescription now lists "stale PVCs/PVs" under Efficiency; the short wording keeps the tool catalog withinTestToolCatalogContextBudgetafter main's growth.releasedPVcounts Bound PVs as passes; Available and Pending PVs are not counted at all.missingInputskeys:pvc-consumers,pv-deletion-events,pv-release-time, pluspersistentvolumeclaims/persistentvolumeswhen those inventories are unavailable. They appear only when a claim or volume went unevaluated, so Radar's "Some checks could not run" banner and Hub's "checks skipped" copy stay accurate.pkg/auditchanges are additive: newCheckInputfields and one exported predicate,PVNeedsReclaimEvents. Radar Hub importspkg/checks, notpkg/audit, and keys checks by ID (titles are display-only), so an older Hub renders the new checks from the cluster's catalog. No new endpoint, so no feature flag is needed.Known limits (product calls, not bugs)
pkg/checks.sortEffectiveFindings), not by size, so a large disk can sit behind "View all". Size-first ordering needs a field on the sharedpkg/checksfinding that Hub would also carry; it is a follow-up.pvcLongPendingstays Medium like the other two checks. A Pending claim has no disk yet, but the detector can only emit High or Medium; Low is an org override in Hub.data-kafka-4after a 5â3 scale-down stays silent.claim-<user>after culling, CI caches and backup targets mounted by short-lived Pods, KubeVirt VMs with direct PVC volumes, and snapshot sources or clone operations beyond the observed Pending PVC references. Ignored namespaces or disabling the check are the escape hatches.