Skip to content

Cluster Audit: find unclaimed storage (unconsumed PVCs, long-Pending claims, Released PVs) - #2055

Merged
nadaverell merged 13 commits into
mainfrom
feat/storage-waste-audit-checks
Oct 12, 2026
Merged

nadaverell merged 13 commits into
mainfrom
feat/storage-waste-audit-checks

Conversation

@nadaverell

@nadaverell nadaverell commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

In a real cleanup every piece of unused storage was found by hand:

  • about 311Gi of Loki and Mimir disks left behind after their apps were deleted
  • about 45Gi of 2.7-year-old PVCs in default
  • an 8Gi Postgres PVC with no workload
  • a PVC that survived its Helm uninstall
  • two PVCs Pending for about three years on a WaitForFirstConsumer storage class, which the issue detector deliberately ignores
  • Released PVs stuck in a VolumeFailedDelete retry loop

Cluster Audit now reports these as snapshot Efficiency checks (Medium on the existing Checks and MCP get_cluster_audit surfaces):

Check Reports
pvcNoConsumer Bound PVCs older than a day with no consumer Radar can observe: Pods, Deployments, ReplicaSets, StatefulSets (including zero replicas and leftover ordinals after a scale-down), DaemonSets, Jobs, CronJobs, generic ephemeral volumes, and Pending PVC clones (a clone protects its source only while it is Pending). On Kubernetes 1.37+, the claim's Unused condition is used when present (below). Shows size, storage class, age, bound PV and reclaim policy.
pvcLongPending PVCs Pending for over 24h with no observed consumer, WaitForFirstConsumer included. Shown as "Pending since creation"; a claim never returns to Pending. A Pending claim that has a consumer is already an outage issue, so it isn't double-reported as waste.
releasedPV Released PVs: those kept by a Retain policy (after a 24h grace when release time is known, or flagged with "release time unknown" and PV age when it is not), and those whose deletion hasn't completed after an hour of known release duration. A recent VolumeFailedDelete warning establishes deletion failure immediately, even without a phase-transition timestamp. Delete PVs without that timestamp or a recent warning are not evaluated. Failed PVs (an in-tree delete or recycle failed after the claim was deleted) are reported immediately, with the latest current VolumeFailedDelete/VolumeFailedRecycle warning or else the PV's status.message.

Not waste by construction:

  • Owned PVCs are skipped. A PVC with a controller owner reference, such as a hibernated CloudNativePG cluster's volumes, a stopped KubeVirt VM's disks, or workflow volumeClaimTemplates, follows its owner's lifecycle, so these checks leave it to the owner.
  • The check says "no consumer observed", never "unused" or "safe to delete". The caveat about consumers Radar can't see (CRDs mounting volumes intermittently, snapshot operations) lives in the check description (shown once per check) and the remediation (returned with every MCP finding), not on every row. Rows state facts only: size, class, age, bound PV, reclaim policy, and the Unused time or binding-mode evidence where present.
  • Remediation says how to reclaim. No consumer: if nothing needs the data, delete the claim; if its PV is Retain, reclaim that too. Retain PVs: deleting the PV keeps its disk; for a dynamically provisioned volume set the reclaim policy to Delete, otherwise delete the PV and then the disk in your cloud provider. Both keep the "confirm nothing needs the data" step first.

Kubernetes 1.37 Unused condition (PVC protection controller, beta, on by default):

  • Unused=False (a running Pod uses the claim) counts as a consumer, even if Radar's Pod inventory missed it.
  • With Unused=True, the 24h floor counts from when the claim became unused instead of from its creation, so a claim whose workload was deleted minutes ago isn't flagged. The finding adds "Kubernetes reports it unused by any Pod since (5d ago)". It reports the condition rather than promising a minimum idle time: upstream notes the time can be off in either direction because of controller delays. Right after the feature is enabled (e.g. an upgrade to 1.37), every unused claim's timestamp is the enable time, so long-unused claims are held back for a day and then reported with that shorter age.
  • No condition or Unknown keeps the creation-age behavior; absence never implies unused. pvcLongPending doesn't use it.

Deleting claims and volumes → stuckTerminating:

  • PVCs and PVs are now stuckTerminating subjects, with the same 5m/30m ramp as other kinds. Deleting PVCs were already skipped by the storage checks; deleting PVs are now skipped by releasedPV too, so a deleting Bound PV no longer counts as a pass and a deleting Released PV isn't reported twice.
  • For kubernetes.io/pvc-protection and kubernetes.io/pv-protection, the finding explains what holds the deletion and what happens when it's released (the claim is deleted and the volume follows its reclaim policy), suggests a backup if the deletion was unintended, and says not to remove the finalizer. It never tells the user to stop Pods or delete the claim: a deletion can't be cancelled, and completing it can delete the data.
  • The shared stuckTerminating remediation (shown in Checks and MCP) now exempts storage finalizers from its "clear finalizers as a last resort" advice.

Coverage:

  • Inputs are gated on cache readiness. PVs and StorageClasses are cluster-scoped and checked against the caller's exact grants per kind; PVCs, workload templates, and events follow namespace visibility.
  • missingInputs keeps its existing meaning: something was not evaluated. A storage input is recorded only when its absence left a claim or volume unevaluated, so unreadable evidence never looks like a clean pass:
    • an unreadable PVC inventory (persistentvolumeclaims)
    • unknown consumers (pvc-consumers)
    • a PV inventory the caller may read but Radar couldn't list (persistentvolumes)
    • a Released Delete volume still in its 1h grace period whose outcome depends on events Radar can't see (pv-deletion-events)
    • a Released Delete volume with a nil or zero phase-transition timestamp and no recent deletion warning: release duration is unknown, so PV creation age cannot establish a stalled deletion or a pass (pv-release-time)
  • Gaps that still let a finding fire are stated in the finding instead: "reclaim policy not visible", "StorageClass binding mode not visible", "Deletion warning events not fully visible". Radar's "Some checks could not run" banner and Hub's "checks skipped" copy stay accurate, so no consumer change is needed.
  • Without a PV grant, Released PVs aren't subjects of that caller's scan, so namespace-restricted users don't get a permanent missing input.
  • Failed PVs never pass and are never skipped. The PV controller sets Failed only when reclaiming failed, so the phase alone establishes the finding. Events only add detail; incomplete coverage adds the "Deletion warning events not fully visible" caveat, never pv-deletion-events. The controller emits the warning once, at the transition, and events expire (1h by default), so status.message is the fallback.
  • What releasedPV counts: Bound PVs pass, Released and Failed PVs are judged, and Available/Pending PVs are not counted. They have no claim to have outlived, so counting them as passes would only inflate the pass rate (for example, a local-static-provisioner pool).

No Radar-kept usage history, cost column or delete action; those are follow-ups. pkg/audit changes are additive.

Testing

  • Table tests cover Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, CronJobs, generic ephemeral volumes, and Pending clones through both reference fields (including owned, young, and deleting destinations). Cross-namespace dataSourceRef references protect only the source in the referenced namespace. Bound clones, non-core references, and VolumeSnapshot references do not protect a source. Tests also cover StatefulSet replicas 0 and leftover ordinals, controller-owned (CNPG) claims, the 24h age floor at a fixed clock, WaitForFirstConsumer with and without consumers, and Retain vs Delete with and without events. Nil and zero release timestamps cover Delete skips, immediate recent-warning findings, and Retain findings with unknown release time and PV age. They also cover missing inputs only for skipped subjects (otherwise the caveat is in the finding), PV inventories that are unavailable vs. outside the caller's grant, and unreadable namespaces suppressing findings.
  • Unused condition: False with no observed consumer and with incomplete inventory, True 2h ago on a 30-day-old claim, the exact 24h boundary, future and zero timestamps, True 10 days ago with the evidence sentence, zero-replica StatefulSet and Pod consumers, absent and Unknown conditions, and Pending claims ignoring it.
  • Audit counts, MCP audit scope (including pv-release-time appearing only with a PV grant), and the ChecksView input labels.
  • Failed PVs: a Delete PV with a current warning (event text and age), with only status.message, with partial or unreadable events (caveat, evaluated, not passed, no missing input), with a warning older than the Failed transition (ignored), Recycle with VolumeFailedRecycle, and a policy switched to Retain after the failure. Counts: Bound passes; Failed is evaluated but not passed; Available and Pending are not counted. internal/audit collects events for Failed PVs.
  • make test and make tsc pass. The original change was checked live on a kind cluster (an unconsumed bound PVC and a Released Retain PV through REST and MCP).

Live verification (kind, Kubernetes v1.36.1 so lastPhaseTransitionTime is populated; real binary in local and proxy-auth modes; REST + MCP + Checks UI)

  • Earlier round (independent live tester, head 3dc5f6a3):
    • The prior proof re-run passed.
    • Unreadable StorageClasses produce the "binding mode not visible" caveat and no missing input.
    • A namespace user without a PV grant gets no persistentvolumes missing input.
    • A real skip (no Pod list) is listed as pvc-consumers.
    • A Released Delete PV past its grace period is flagged, with the "not fully visible" caveat under incomplete events.
    • A CSI VolumeFailedDelete on a Released PV is reported as "deletion is failing".
    • Warnings from before the current release are ignored.
    • The Checks view shows the "could not run" banner only on real skips.
    • MCP get_cluster_audit matches REST.
    • The WaitForFirstConsumer, default-class and class-not-found messages (readable StorageClasses) are covered by unit tests only.
  • That round found this bug: a hostPath Delete PV whose in-tree deletion failed went to Failed with a real VolumeFailedDelete event, and was counted as a releasedPV pass.
  • After this fix (head 4be95ee8: the binary was built from 3dc5f6a3 plus the uncommitted fix, whose diff is identical to the 4be95ee8 commit; same repro: hostPath Delete PV at /var/..., claim deleted, PV Failed with status.message set and one event in default):
    • Admin view, releasedPV evaluated 2, passed 1 (the Bound PV passes, the Failed PV is flagged): Failed PV: reclaiming the volume failed after its claim was deleted (reclaim policy Delete). Latest reclaim warning (2m ago): host_path deleter only supports /tmp/.+ but received provided /var/fix2055-delete-fail. Capacity 1Mi; storage class ""; former claim fix2055-f3/claim-delete-fail; Failed for 2m. MCP shows the same text.
    • A proxy-auth user with a PV grant who sees only the fixture namespace, so the event in default is hidden: the same finding, carrying the Status message: host_path deleter only â€Ķ text plus "Deletion warning events not fully visible". There is no pv-deletion-events missing input, and REST and MCP match.
    • The Checks UI renders the finding under "Released or Failed persistent volume".
  • Documentation commits and the README check-count test followed those live checks. The release-time uncertainty and Pending-clone scenarios have unit and MCP-handler coverage; their live verification remains pending.

Live verification on GKE 1.37.1 (nonprod control plane upgraded to 1.37.1; current head)

  • The controller set Unused=False on a claim with a running Pod and Unused=True (timestamped at Pod deletion) on two claims whose Pods were deleted. A standard-rwo (WaitForFirstConsumer) clone of one of them stayed Pending.
  • A snapshot of those live objects ran through the audit with creation and unused times shifted back to clear the 24h floor:
Variant Result
As captured (claims minutes old) No storage findings; only the pre-existing in-use claim evaluated, passing
Created 10d ago, unused 5d orphan flagged with the Unused time (5d ago); clone-source protected by the Pending clone; in-use passes; the abandoned Pending clone flagged as pvcLongPending
Same, in-use Pod removed from the snapshot in-use still passes on Unused=False
Created 10d ago, unused since minutes ago orphan and clone-source held back (not evaluated)
Same as row 2, clone removed clone-source flagged
  • Deleting PVC/PV routing, protection-finalizer copy (including that it never advises completing the deletion) and the shared remediation's storage exception are covered in pkg/audit tests; all three fail on the previous head.
  • make test, make tsc and the audit/MCP suites pass on the current head, after merging main (which brought Detect custom resources stuck deleting on an operator's finalizer #2053).

Live check of stuck deletion (GKE 1.37.1, head 9ea8f6871): a PVC still mounted by a running Pod was deleted, then its bound PV. After 6 minutes both were reported by stuckTerminating as warnings: the PVC held by kubernetes.io/pvc-protection, the PV by kubernetes.io/pv-protection plus the two CSI finalizers, each with the protection explanation. releasedPV counted only the cluster's other (Bound) PV, so the deleting PV was neither passed nor double-reported. Deleting the namespace then removed the PVC, PV and disk.

Docs

Rollout and compatibility

  • The MCP get_cluster_audit description now lists "stale PVCs/PVs" under Efficiency; the short wording keeps the tool catalog within TestToolCatalogContextBudget after main's growth.
  • The three checks are on by default at Medium, so posture counts (Medium findings, per-check pass rates) shift on upgrade for any cluster with unclaimed PVCs or Released/Failed PVs. Each check can be disabled in audit settings.
  • releasedPV counts Bound PVs as passes; Available and Pending PVs are not counted at all.
  • New missingInputs keys: pvc-consumers, pv-deletion-events, pv-release-time, plus persistentvolumeclaims / persistentvolumes when those inventories are unavailable. They appear only when a claim or volume went unevaluated, so Radar's "Some checks could not run" banner and Hub's "checks skipped" copy stay accurate.
  • pkg/audit changes are additive: new CheckInput fields and one exported predicate, PVNeedsReclaimEvents. Radar Hub imports pkg/checks, not pkg/audit, and keys checks by ID (titles are display-only), so an older Hub renders the new checks from the cluster's catalog. No new endpoint, so no feature flag is needed.

Known limits (product calls, not bugs)

  • Findings within a check are ordered by severity, then kind, namespace and name (pkg/checks.sortEffectiveFindings), not by size, so a large disk can sit behind "View all". Size-first ordering needs a field on the shared pkg/checks finding that Hub would also carry; it is a follow-up.
  • pvcLongPending stays Medium like the other two checks. A Pending claim has no disk yet, but the detector can only emit High or Medium; Low is an org override in Hub.
  • StatefulSet scale-down leftovers are not flagged: every claim-template ordinal counts as consumed, so data-kafka-4 after a 5→3 scale-down stays silent.
  • Consumers Radar can't see cause false positives, each worded "no consumer observed" with the CRD caveat. Examples: JupyterHub claim-<user> after culling, CI caches and backup targets mounted by short-lived Pods, KubeVirt VMs with direct PVC volumes, and snapshot sources or clone operations beyond the observed Pending PVC references. Ignored namespaces or disabling the check are the escape hatches.
  • PVC subjects follow namespace visibility, as other namespaced audit subjects do. Like those kinds, PVCs in a namespace that a namespace-scoped informer doesn't cover are not reported as a missing input.

@nadaverell
nadaverell requested a review from hisco as a code owner October 10, 2026 03:26
@cursor

cursor Bot commented Oct 10, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

Add storage waste checks to Cluster Audit

âœĻ Enhancement 🧊 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

â€Ē Add snapshot checks for unconsumed Bound PVCs, long-Pending PVCs, and Released PVs.
â€Ē Gate absence findings on readable consumer inventories and scope storage evidence to caller
 grants.
â€Ē Surface cautious findings and missing-input labels in Checks and MCP, with tests and
 documentation.
Diagram

graph TD
  Cache["Resource cache"] --> Collector["Storage collector"] --> Checks["Storage checks"] --> Results["Audit results"] --> Surfaces["Checks and MCP"]
  Grants["Caller grants"] --> Collector
  Consumers["Workload references"] --> Checks
Loading
High-Level Assessment

The additive snapshot checks fit the existing cache-backed audit and avoid implying that unobserved consumers make storage safe to delete. Durable usage history or deletion automation would require different evidence and safeguards; keeping those out of this PR is appropriate.

Files changed (17) +1010 / -7

Enhancement (8) +390 / -0
runner.goPass scoped storage inventories into audit runs +9/-0

Pass scoped storage inventories into audit runs

â€Ē Collects storage inputs before typed consumer inventories and attaches PVCs, PVs, StorageClasses, events, and coverage metadata to the check input.

internal/audit/runner.go

scope.goResolve exact PV and StorageClass list grants +5/-0

Resolve exact PV and StorageClass list grants

â€Ē Includes both cluster-scoped storage kinds in read-scope authorization checks independently of dynamically watched resources.

internal/audit/scope.go

storage.goCollect readiness- and grant-gated storage evidence +64/-0

Collect readiness- and grant-gated storage evidence

â€Ē Collects PVC subjects and eligible cluster-scoped inventories, records namespaces with complete consumer coverage, and reads deletion events only when a Released Delete PV needs them.

internal/audit/storage.go

ChecksView.tsxLabel storage evidence gaps in Checks +3/-0

Label storage evidence gaps in Checks

â€Ē Adds readable labels for missing PVC consumers, binding mode, and PV deletion events.

packages/k8s-ui/src/components/checks/ChecksView.tsx

checks.goRun storage checks in the audit pipeline +1/-0

Run storage checks in the audit pipeline

â€Ē Invokes the new snapshot storage evaluation alongside existing audit checks.

pkg/audit/checks.go

registry.goRegister three Efficiency storage checks +26/-0

Register three Efficiency storage checks

â€Ē Adds check descriptions, cautious remediation guidance, and Kubernetes references for unconsumed PVCs, long-Pending PVCs, and Released PVs.

pkg/audit/registry.go

storage.goEvaluate unclaimed PVCs and Released PVs +270/-0

Evaluate unclaimed PVCs and Released PVs

â€Ē Matches claims against Pods and built-in workload templates, excludes controller-owned and young claims, and evaluates Released PVs using reclaim policy, age, and UID-matched deletion warnings. Reports missing evidence without treating unreadable inventories as proof of absence.

pkg/audit/storage.go

types.goExtend audit input with storage and coverage fields +12/-0

Extend audit input with storage and coverage fields

â€Ē Adds PVC, PV, StorageClass, and event inventories plus explicit consumer-namespace and deletion-event completeness metadata.

pkg/audit/types.go

Tests (5) +574 / -3
scope_test.goAdjust scope tests for storage grant checks +6/-3

Adjust scope tests for storage grant checks

â€Ē Accounts for the additional PV and StorageClass authorization calls while preserving existing scope assertions.

internal/audit/scope_test.go

storage_test.goTest scoped storage collection and cache coverage +157/-0

Test scoped storage collection and cache coverage

â€Ē Covers exact cluster grants, unreadable or unsynced inventories, namespace-limited consumer evidence, partial event visibility, and conditional event collection.

internal/audit/storage_test.go

audit_scope_test.goTest MCP storage findings under PV grants +41/-0

Test MCP storage findings under PV grants

â€Ē Verifies that denying PV access removes PV findings and reclaim metadata without suppressing an otherwise supported PVC finding or corrupting counts.

internal/mcp/audit_scope_test.go

counts_test.goAccount for storage inputs in completeness tests +5/-0

Account for storage inputs in completeness tests

â€Ē Updates missing-input expectations and supplies present-but-empty storage inventories when testing a complete scan.

pkg/audit/counts_test.go

storage_test.goCover storage eligibility, consumers, and deletion evidence +365/-0

Cover storage eligibility, consumers, and deletion evidence

â€Ē Tests workload families, retained StatefulSet ordinals, owner exclusions, age boundaries, binding modes, missing inputs, PV counts, and warning-event identity and timing.

pkg/audit/storage_test.go

Documentation (4) +46 / -4
README.mdList storage findings under Cluster Audit efficiency +2/-2

List storage findings under Cluster Audit efficiency

â€Ē Adds the PVC and PV checks to the feature overview and removes the outdated fixed check count.

README.md

configuration.mdDocument storage check eligibility and evidence limits +41/-0

Document storage check eligibility and evidence limits

â€Ē Explains thresholds, consumer coverage, owner exclusions, storage grants, missing inputs, and the distinction between observed absence and deletion safety.

docs/configuration.md

mcp.mdDocument storage audit authorization in MCP +1/-1

Document storage audit authorization in MCP

â€Ē Clarifies PV subject grants and how incomplete consumer, storage metadata, or event evidence affects audit results.

docs/mcp.md

tools.goAdvertise storage checks in the MCP audit tool +2/-1

Advertise storage checks in the MCP audit tool

â€Ē Extends the tool description to mention PVC consumer, long-Pending claim, and Released PV findings.

internal/mcp/tools.go

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) ðŸŽĻ UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Recent volumes pass without event access ✓ Resolved
Description
checkStorage records a Released/Delete volume as evaluated, then exits during its one-hour grace
period before recording the missing pv-deletion-events input. When event coverage is unsynced or
limited by the caller's namespace grants, a deletion warning may be unavailable, but the volume
still counts as passed with no indication that evidence is missing.
Code

pkg/audit/storage.go[R115-116]

+				if releasedAt.IsZero() || now.Sub(releasedAt.Time) <= pvDeleteGrace {
+					continue
Evidence
The collector can leave Events unavailable or mark deletion-event coverage incomplete for a scoped
caller. The check records every PV at line 95, but its recent-volume exit at lines 115–116 precedes
the missing-input append at lines 123–125; check counts derive passes from recorded evaluations
without findings.

internal/audit/storage.go[53-63]
pkg/audit/storage.go[93-125]
pkg/audit/checks.go[1682-1725]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A recently Released Delete volume is counted as passing when deletion events are unavailable, without reporting the missing event input.
## Fix Focus Areas
- pkg/audit/storage.go[93-125]
- internal/audit/storage.go[53-63]
## Recommended Fix
Check event completeness before the one-hour early exit. Report `pv-deletion-events` when warning evidence is unavailable, and do not count a recent volume as passing unless its outcome can be established.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Reused volumes show stale delete alarms ✓ Resolved
Description
checkStorage accepts a recent VolumeFailedDelete event for a PV's UID without comparing its
timestamp to the PV's current LastPhaseTransitionTime. If a volume is reclaimed, rebound and
released again within 24 hours, a warning from the earlier release produces an immediate failure
finding for the new release.
Code

pkg/audit/storage.go[R110-113]

+		case corev1.PersistentVolumeReclaimDelete:
+			failure, hasFailure := deleteFailures[pv.UID]
+			if hasFailure && now.Sub(failure.at) <= pvDeleteEventWindow {
+				message = "Released PV deletion is failing: " + failure.event.Message
Evidence
The check reads the current phase-transition time for grace-period decisions, but its
immediate-failure branch checks only whether the UID-matched warning is within 24 hours of now. The
event matcher also selects warnings by UID and event time without checking the current release
transition.

pkg/audit/storage.go[99-116]
pkg/audit/storage.go[237-267]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A deletion warning from an earlier release can be attributed to a PV's current Released phase.
## Fix Focus Areas
- pkg/audit/storage.go[99-113]
- pkg/audit/storage.go[237-267]
## Recommended Fix
When `LastPhaseTransitionTime` is present, require a deletion warning to fall within the current Released period before using it as immediate failure evidence. Preserve the documented fallback when that transition time is unavailable, and test a rebound-and-released PV with an older warning.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Long-Pending claims wrongly flag missing binding-mode data ✓ Resolved
Description
In checkStorage's Pending branch, the class is looked up only when StorageClassName is non-nil.
If the name is nil, or names a class that is missing from a fully readable StorageClass list, sc
stays nil, pvc-binding-mode is appended to missingInputs, and the message says "StorageClass
binding mode not visible". Users who can read every StorageClass then get a permanent "some checks
could not run" notice for those claims. It also hides the more likely cause of the stuck claim: the
class it names does not exist, or no class or default class applies to it.
Code

pkg/audit/storage.go[R75-85]

+			if pvc.Spec.StorageClassName == nil || *pvc.Spec.StorageClassName != "" {
+				var sc *storagev1.StorageClass
+				if pvc.Spec.StorageClassName != nil {
+					sc = classes[*pvc.Spec.StorageClassName]
+				}
+				if sc == nil {
+					if input.StorageClasses == nil {
+						tr.missingInputs = append(tr.missingInputs, "storageclasses")
+					}
+					tr.missingInputs = append(tr.missingInputs, "pvc-binding-mode")
+					message += " StorageClass binding mode not visible."
Evidence
The class is resolved only when pvc.Spec.StorageClassName != nil (lines 77-79). Whenever `sc ==
nil, the code adds pvc-binding-mode (line 84), even when input.StorageClasses` is non-nil,
meaning the inventory was read in full. The storageclasses missing input is correctly guarded by
input.StorageClasses == nil (line 81), but pvc-binding-mode has no such guard. The test case
"unknown class no consumer" only covers StorageClasses = nil, so neither the nil-name case nor the
class-not-found case is tested. The PR description and docs say missing inputs should appear only
when evidence is actually unavailable, so namespace-restricted users don't see a permanent warning.

pkg/audit/storage.go[75-89]
docs/configuration.md[742-749]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
In `checkStorage` (`pkg/audit/storage.go`), a long-Pending PVC whose `storageClassName` is nil, or names a class absent from a fully readable StorageClass inventory, always adds `pvc-binding-mode` to `missingInputs` and says the binding mode is "not visible", even though the evidence was read.
## Fix Focus Areas
- pkg/audit/storage.go[75-89]
## Recommended Fix
Add `pvc-binding-mode` (and the "not visible" text) only when `input.StorageClasses == nil`. When StorageClasses were read and a named class is not found, add a message such as " StorageClass %q not found." and do not add a missing input. When `storageClassName` is nil and classes are readable, look up the default class (annotation `storageclass.kubernetes.io/is-default-class=true`); if there is none, say no default StorageClass applies. Add table tests for the nil-name and class-not-found cases with readable StorageClasses.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (1)
4. Cloud checks falsely appear skipped ✓ Resolved
Description
checkStorage adds pv-deletion-events to missingInputs even after evaluating releasedPV and
emitting its finding. When deletion-event coverage is partial, Radar Hub Web receives that key
through Hub and tells users checks were skipped; the new PVC metadata inputs can produce the same
mismatch.
Code

pkg/audit/storage.go[R123-125]

+			if input.Events == nil || !input.PVDeletionEventsComplete {
+				tr.missingInputs = append(tr.missingInputs, "pv-deletion-events")
+			}
Evidence
Radar emits the deletion-event missing-input key before appending the Released PV finding. Hub
forwards the key unchanged, while Hub Web renders missing inputs as checks that were skipped.

radar -> radar-hub
radar -> radar-hub-web
pkg/audit/storage.go[123-138]
internal/audit/storage.go[59-63]
External repo: skyhook-dev/radar-hub, internal/server/fleet_handlers.go [1390-1398]
External repo: skyhook-dev/radar-hub-web, src/pages/fleet/checks/ChecksQueue.tsx [244-250]
External repo: skyhook-dev/radar-hub-web, src/pages/home/PostureCards.tsx [270-280]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Storage checks now report missing supplemental evidence through `missingInputs` even when the checks ran. Radar Hub Web interprets every missing input as a skipped check.
## Fix Focus Areas
- pkg/audit/storage.go[123-125]
- /cross_repos/radar-hub-web/src/pages/fleet/checks/ChecksQueue.tsx[244-250]
- /cross_repos/radar-hub-web/src/pages/home/PostureCards.tsx[270-280]
## Recommended Fix
Coordinate with radar-hub-web to distinguish supplemental evidence gaps from skipped checks in its coverage messages. Preserve the evidence warning without claiming that the storage checks did not run.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
ðŸ’Ą Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread pkg/audit/storage.go Outdated
Comment thread pkg/audit/storage.go Outdated
Comment thread pkg/audit/storage.go
Comment thread pkg/audit/storage.go Outdated
@cursor

cursor Bot commented Oct 10, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

2 similar comments
@cursor

cursor Bot commented Oct 10, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cursor

cursor Bot commented Oct 11, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

nadaverell added a commit that referenced this pull request Oct 11, 2026
â€Ķ volumes that can land on Hyperdisk-only nodes (#2069)

## Why

A prod Redis StatefulSet was rescheduled onto an `n4d-standard-4` node
after a node-pool change. N4D machines attach only Hyperdisk, and
Redis's volumes are `premium-rwo` (pd-ssd), so the volume never attached
and the pods stayed in ContainerCreating for hours.

Radar reported the attach failure, but with the wrong event. `diagnose`
and `issues` both showed:

> `ControllerPublish not permitted on node "â€Ķ6uw0" due to backoff
condition`

That is the CSI attacher's retry notice. The event that names the cause
was older and was dropped, because post-bind detection keeps only the
latest event per pod:

> `googleapi: Error 400: pd-ssd disk type cannot be used by
n4d-standard-4 machine type., badRequest`

## What changes

**Diagnose and issues (post-bind detection)**
- When the latest attach event is a backoff echo, report the newest
informative attach error **for the same pod UID and the same volume**.
Onset is taken from the first real failure.
- Parse Compute Engine's disk/machine incompatibility into a plain
`Cause` and `Action`: "pd-ssd volumes can't attach to n4d-standard-4
nodes: the N4D machine series supports only Hyperdisk". The action
offers two fixes: pin the workload to a compatible pool, or move it to a
Hyperdisk StorageClass that every eligible pool supports.
- When only echoes remain (the real event has expired), derive the same
cause from the node's `node.kubernetes.io/instance-type` and the PV's
provisioned disk type.
- `diagnose`'s `startupBlockers` now carry `cause` and `action`. Both
are optional additions.
- Events recorded against a predecessor pod's UID are ignored.
StatefulSet pods are recreated under the same name, so without this a
dead pod's error could be shown for its replacement.

**Cluster Audit: new check `pdVolumeHyperdiskOnlyNode` (Reliability)**
- Flags a workload when it mounts a GCE Persistent Disk volume (`pd-*`)
and its pods could be scheduled onto a node from a Hyperdisk-only
machine series.
- Constraints come from the **owner's current pod template**
(Deployment, StatefulSet or DaemonSet). Node selector, required node
affinity, volume node affinity and zone/region labels (including beta PV
labels matched to stable node labels), and NoSchedule/NoExecute taints
are all evaluated. Inter-pod affinity and topology spread are not, and
the message says only what was checked.
- Severity is **danger** when a pod is already on such a node, and
**warning** when one could land there.
- The disk type is trusted only for dynamically provisioned PVs
(`pv.kubernetes.io/provisioned-by`). A static PV that merely borrows a
class name is skipped. StorageClass parameter keys are matched
case-insensitively, as the PD CSI driver matches them.
- Evidence is gated per reader on list access to `nodes`,
`persistentvolumes` and `storageclasses`. A reader missing any of them
gets no evidence, and the check is silent.
- Hyperdisk-only series come from a fixed table based on Compute
Engine's disk-support matrix: A4/A4X, C4/C4A/C4D/C4N, G4, H4D, M4/M4N,
N4/N4A/N4D, X4/X5, Z4D. Series with partial PD support (C3, C3D, H3) and
unknown series are never flagged, so the table can only under-report.

New package `pkg/gkedisk` holds the shared knowledge.
`pkg/audit.CheckInput` gains a `PDPlacement` field. Both are additive
for Hub.

## Decisions

- **Accepted disclosure.** The cache fallback (only backoff echoes
remain) reads the Node, PV and StorageClass with Radar's identity. A
reader who can see the pod's namespace but can't list those kinds will
see the machine type and disk type in the issue's cause. The types are
low-sensitivity, and the fallback only applies once the informative
event has expired. The audit check, by contrast, is gated per reader.
- **Overlap with #2055.** Both PRs touch `internal/audit/scope.go`,
`internal/audit/runner.go`, `pkg/audit/{types,registry,checks}.go`. This
PR's edits sit in separate hunks, and `PDPlacement` is its own field
rather than the PV/PVC/StorageClass inputs #2055 adds. Expect at most a
mechanical rebase.

## Scenario ledger

| Scenario | Expected | Proof |
|---|---|---|
| Real GCE error older than 10 min, latest event is a backoff echo |
Message shows the real error; Cause/Action set; onset from first failure
| `TestDetectPostBindProblems_AttachBackoffShowsRealCause` (fails on
main) |
| Only echoes remain; node n4d, PV pd-ssd | Cause derived from cache |
`â€ĶAttachBackoffOnlyDerivesCause` (fails on main) |
| Echo on a PD-capable node | No cause invented |
`â€ĶAttachBackoffOnCompatibleNode` |
| Predecessor pod UID's events, old or recent | Not used for the
replacement | `â€ĶAttachIgnoresPredecessorPodEvents` |
| Two volumes; echo for A, real error for B | A keeps its own echo |
`â€ĶAttachEchoMatchesItsOwnVolume` |
| Unrelated attach error (timeout) | No disk cause |
`â€ĶUnrelatedAttachFailureHasNoDiskCause` |
| Unpinned STS on pd-ssd, cluster has N4D | Warning on the StatefulSet |
`TestPDPlacement_UnpinnedStatefulSetReachesHyperdiskOnlyNode`,
`TestRunFromCacheFlagsPDVolumeReachingHyperdiskOnlyNode` |
| Pod already on N4D | Danger | `â€ĶPodAlreadyOnHyperdiskOnlyNodeIsDanger`
|
| nodeSelector / NotIn affinity / untolerated taint | Evaluated, passes
| `â€ĶConstraintsThatKeepPodsOff` |
| Zonal disk, N4D only in another zone (nodeAffinity or zone labels) |
No finding | `â€ĶVolumeZoneKeepsPodOffOtherZones`,
`â€ĶLegacyVolumeZoneLabel` |
| PV has beta zone/region labels, node has stable labels | Matching
nodes remain eligible; mismatches are excluded |
`â€ĶLegacyVolumeZoneLabel` |
| Template pin added or removed vs running pod | Template wins |
`â€ĶConstraintsComeFromOwnerTemplate` |
| Mixed-case StorageClass `Type: hyperdisk-balanced` | Classified as
Hyperdisk, never as the PD default | `TestDiskType` |
| Static PV, Hyperdisk volume, non-GCE storage, no N4D nodes | Not
evaluated | `TestDiskType`, `TestPDPlacement_NotApplicable` |
| Reader can't list StorageClasses or nodes | No evidence, no finding |
`TestReadScopeResolvesPDPlacementGrants`, denied-scope case in the
runner test |

## Verification

- `make tsc` passes.
- `make test` passes.
- `cd pkg && go test ./audit ./gkedisk` passes, including matching and
mismatching beta-PV/stable-node zone and region cases.
- `make build` passes (frontend, embed, Go binary).
- Visual-test: skipped (backend diagnostics and audit logic; no rendered
UI changes).
Every piece of unused storage in a real cleanup was found by hand: hundreds
of GiB of disks left by deleted apps, claims Pending for years on
WaitForFirstConsumer classes (which the issue detector deliberately
ignores), and Released volumes stuck failing deletion. Cluster Audit now
reports, as a snapshot:

- Bound PVCs with no consumer Radar can observe (pods, every built-in
  workload template including zero-replica StatefulSets, generic ephemeral
  volumes), with size, class, age, PV and reclaim policy;
- PVCs Pending over 24 hours, including WaitForFirstConsumer claims only
  when nothing references them;
- Released PVs kept by a Retain policy, and those whose deletion is failing
  (a recent VolumeFailedDelete event, with its message).

Findings say "no consumer observed", never "safe to delete". Inputs are
gated on cache readiness and the caller's exact grants, so unreadable
inventories report missing inputs instead of inventing absence findings.
PVCs owned by a controller (a hibernated CloudNativePG cluster's volumes,
a stopped KubeVirt VM's disks, workflow volumeClaimTemplates) are skipped,
since their owner manages their lifecycle; both PVC checks also wait until
a claim is a day old so apply order and restores don't raise findings.
Pending claims are reported only with no observed consumer, as "Pending
since creation" (a claim never returns to Pending). Released Delete volumes
that haven't been deleted after an hour are reported even without events;
retained ones get a 24-hour grace. Missing PV or StorageClass access is
reported only when a claim needed it, every readable PV counts toward the
check, events are listed only when needed, and the new audit inputs have
readable labels.
A recent Released volume no longer counts as passing when deletion events
can't be read, a reused PV name doesn't pick up a stale deletion warning
from an earlier volume (events are matched by UID), and a readable but
absent StorageClass no longer reports binding mode as missing evidence.
Every consumer reads a missing input as "some checks could not run". The
storage checks also listed supplemental evidence gaps there (StorageClass
binding mode, PV reclaim policy, deletion warnings) for claims and volumes
they still evaluated and flagged.

A missing input is now recorded only where the gap left a subject
unevaluated: an unreadable PVC inventory, unknown consumers, an
unavailable PV inventory, or a recently Released Delete volume whose
outcome depends on events Radar can't see. Gaps that still let a finding
fire are stated in its message, including a new "Deletion warning events
not fully visible" caveat. PVs outside a caller's grant are not subjects
of that caller's scan, so they no longer report a missing input.
A PV whose in-tree deletion or recycling fails moves to phase Failed and
stays there; releasedPV only judged Released volumes, so these counted as
passes. Failed volumes are now always evaluated and reported, with the
latest current reclaim warning or else the PV's status message (the
controller records the failure there and emits the event only once).
Incomplete event coverage adds the usual caveat and never skips them.

Available and Pending volumes have no claim to have outlived, so they are
no longer counted as releasedPV passes either.
â€Ķction

README states the Cluster Audit check count again (43, the registry size),
and a test fails when the two drift. The storage review checks section in
docs/configuration.md keeps its contract in half the length, and the MCP
audit access bullet links to it instead of repeating it.
Main added checkPDVolumeHyperdiskOnlyNode while this branch was open, so
the registry now has 44 checks and TestREADMECheckCountMatchesRegistry
needs the README to say so.
@nadaverell
nadaverell force-pushed the feat/storage-waste-audit-checks branch from 7185ace to af625a6 Compare October 12, 2026 07:24
@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@nadaverell
nadaverell merged commit 2590be3 into main Oct 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant