[nadaverell]

Why

A prod Redis StatefulSet was rescheduled onto an n4d-standard-4 node after a node-pool change. N4D machines attach only Hyperdisk, and Redis's volumes are premium-rwo (pd-ssd), so the volume never attached and the pods stayed in ContainerCreating for hours.

Radar reported the attach failure, but with the wrong event. diagnose and issues both showed:

ControllerPublish not permitted on node "…6uw0" due to backoff condition

That is the CSI attacher's retry notice. The event that names the cause was older and was dropped, because post-bind detection keeps only the latest event per pod:

googleapi: Error 400: pd-ssd disk type cannot be used by n4d-standard-4 machine type., badRequest

What changes

Diagnose and issues (post-bind detection)

• When the latest attach event is a backoff echo, report the newest informative attach error for the same pod UID and the same volume. Onset is taken from the first real failure.
• Parse Compute Engine's disk/machine incompatibility into a plain Cause and Action: "pd-ssd volumes can't attach to n4d-standard-4 nodes: the N4D machine series supports only Hyperdisk". The action offers two fixes: pin the workload to a compatible pool, or move it to a Hyperdisk StorageClass that every eligible pool supports.
• When only echoes remain (the real event has expired), derive the same cause from the node's node.kubernetes.io/instance-type and the PV's provisioned disk type.
• diagnose's startupBlockers now carry cause and action. Both are optional additions.
• Events recorded against a predecessor pod's UID are ignored. StatefulSet pods are recreated under the same name, so without this a dead pod's error could be shown for its replacement.

Cluster Audit: new check pdVolumeHyperdiskOnlyNode (Reliability)

• Flags a workload when it mounts a GCE Persistent Disk volume (pd-*) and its pods could be scheduled onto a node from a Hyperdisk-only machine series.
• Constraints come from the owner's current pod template (Deployment, StatefulSet or DaemonSet). Node selector, required node affinity, volume node affinity and zone/region labels (including beta PV labels matched to stable node labels), and NoSchedule/NoExecute taints are all evaluated. Inter-pod affinity and topology spread are not, and the message says only what was checked.
• Severity is danger when a pod is already on such a node, and warning when one could land there.
• The disk type is trusted only for dynamically provisioned PVs (pv.kubernetes.io/provisioned-by). A static PV that merely borrows a class name is skipped. StorageClass parameter keys are matched case-insensitively, as the PD CSI driver matches them.
• Evidence is gated per reader on list access to nodes, persistentvolumes and storageclasses. A reader missing any of them gets no evidence, and the check is silent.
• Hyperdisk-only series come from a fixed table based on Compute Engine's disk-support matrix: A4/A4X, C4/C4A/C4D/C4N, G4, H4D, M4/M4N, N4/N4A/N4D, X4/X5, Z4D. Series with partial PD support (C3, C3D, H3) and unknown series are never flagged, so the table can only under-report.

New package pkg/gkedisk holds the shared knowledge. pkg/audit.CheckInput gains a PDPlacement field. Both are additive for Hub.

Decisions

• Accepted disclosure. The cache fallback (only backoff echoes remain) reads the Node, PV and StorageClass with Radar's identity. A reader who can see the pod's namespace but can't list those kinds will see the machine type and disk type in the issue's cause. The types are low-sensitivity, and the fallback only applies once the informative event has expired. The audit check, by contrast, is gated per reader.
• Overlap with Cluster Audit: find unclaimed storage (unconsumed PVCs, long-Pending claims, Released PVs) #2055. Both PRs touch internal/audit/scope.go, internal/audit/runner.go, pkg/audit/{types,registry,checks}.go. This PR's edits sit in separate hunks, and PDPlacement is its own field rather than the PV/PVC/StorageClass inputs Cluster Audit: find unclaimed storage (unconsumed PVCs, long-Pending claims, Released PVs) #2055 adds. Expect at most a mechanical rebase.

Scenario ledger

Scenario	Expected	Proof
Real GCE error older than 10 min, latest event is a backoff echo	Message shows the real error; Cause/Action set; onset from first failure	TestDetectPostBindProblems_AttachBackoffShowsRealCause (fails on main)
Only echoes remain; node n4d, PV pd-ssd	Cause derived from cache	…AttachBackoffOnlyDerivesCause (fails on main)
Echo on a PD-capable node	No cause invented	…AttachBackoffOnCompatibleNode
Predecessor pod UID's events, old or recent	Not used for the replacement	…AttachIgnoresPredecessorPodEvents
Two volumes; echo for A, real error for B	A keeps its own echo	…AttachEchoMatchesItsOwnVolume
Unrelated attach error (timeout)	No disk cause	…UnrelatedAttachFailureHasNoDiskCause
Unpinned STS on pd-ssd, cluster has N4D	Warning on the StatefulSet	TestPDPlacement_UnpinnedStatefulSetReachesHyperdiskOnlyNode, TestRunFromCacheFlagsPDVolumeReachingHyperdiskOnlyNode
Pod already on N4D	Danger	…PodAlreadyOnHyperdiskOnlyNodeIsDanger
nodeSelector / NotIn affinity / untolerated taint	Evaluated, passes	…ConstraintsThatKeepPodsOff
Zonal disk, N4D only in another zone (nodeAffinity or zone labels)	No finding	…VolumeZoneKeepsPodOffOtherZones, …LegacyVolumeZoneLabel
PV has beta zone/region labels, node has stable labels	Matching nodes remain eligible; mismatches are excluded	…LegacyVolumeZoneLabel
Template pin added or removed vs running pod	Template wins	…ConstraintsComeFromOwnerTemplate
Mixed-case StorageClass Type: hyperdisk-balanced	Classified as Hyperdisk, never as the PD default	TestDiskType
Static PV, Hyperdisk volume, non-GCE storage, no N4D nodes	Not evaluated	TestDiskType, TestPDPlacement_NotApplicable
Reader can't list StorageClasses or nodes	No evidence, no finding	TestReadScopeResolvesPDPlacementGrants, denied-scope case in the runner test

Verification

• make tsc passes.
• make test passes.
• cd pkg && go test ./audit ./gkedisk passes, including matching and mismatching beta-PV/stable-node zone and region cases.
• make build passes (frontend, embed, Go binary).
• Visual-test: skipped (backend diagnostics and audit logic; no rendered UI changes).

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Explain incompatible disk attaches and audit risky PD placement

🐞 Bug fix ✨ Enhancement 🧪 Tests 🕐 40+ Minutes

AI Description

• Replace CSI backoff echoes with the matching attach failure and explain incompatible disk types.
• Audit workloads whose Persistent Disks can reach Hyperdisk-only nodes, subject to reader
 permissions.
• Add coverage for event correlation, disk provenance, scheduling constraints, and audit visibility.

Diagram

graph TD
  Cache[("Resource cache")] --> Detector["Post-bind detection"] --> Surfaces["Diagnose and issues"]
  Cache --> Evidence["Scoped evidence"] --> Check["Placement audit"] --> Findings["Reliability findings"]
  Rules["GKE disk rules"] --> Detector & Evidence & Check

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Simulate full Kubernetes scheduling

• ➕ Could account for inter-pod affinity, topology spread, and other scheduler filters.
• ➖ Substantially more complex and costly to maintain against scheduler behavior.
• ➖ Requires broader cluster state and authorization decisions.

Recommendation: Keep the bounded placement check: it evaluates the principal node and volume constraints, states its limits, and keeps cluster-wide evidence permission-gated. A full scheduler simulation is not justified for this targeted incompatibility.

Files changed (15) +1307 / -6

Enhancement (9) +502 / -0

pd_placement.goCollect permission-gated PD placement evidence +77/-0

Collect permission-gated PD placement evidence

• Collects Hyperdisk-only nodes and bound claims’ PVs and disk types from synced, cluster-wide caches. Returns no evidence unless the reader can list nodes, PVs, and StorageClasses.

internal/audit/pd_placement.go

runner.goFeed PD placement evidence into audit checks +1/-0

Feed PD placement evidence into audit checks

• Adds placement evidence collection to the cache-backed audit runner before checks execute.

internal/audit/runner.go

scope.goResolve cluster-resource grants for placement audits +6/-0

Resolve cluster-resource grants for placement audits

• Includes nodes, PVs, and StorageClasses when resolving a reader’s cluster-scoped list permissions.

internal/audit/scope.go

tools_diagnose.goExpose causes and actions in startup blockers +4/-0

Expose causes and actions in startup blockers

• Adds optional cause and action fields to diagnose startup blockers and copies them from post-bind detections.

internal/mcp/tools_diagnose.go

checks.goRun the PD placement reliability check +6/-0

Run the PD placement reliability check

• Registers the new placement evaluation in the audit check runner; absent evidence leaves it silent.

pkg/audit/checks.go

disk_compat.goAssess PD workload reachability to incompatible nodes +265/-0

Assess PD workload reachability to incompatible nodes

• Adds a workload-level check using owner templates, node constraints, volume topology, and taints. Reports a warning for reachable Hyperdisk-only nodes and danger when a PD-backed pod is already placed on one.

pkg/audit/disk_compat.go

registry.goDocument the new reliability check +9/-0

Document the new reliability check

• Adds the check’s title, description, remediation options, and links to disk-support and scheduling references.

pkg/audit/registry.go

types.goAdd optional PD placement input +5/-0

Add optional PD placement input

• Extends CheckInput with a separate, optional evidence field for the new placement check.

pkg/audit/types.go

gkedisk.goShare GKE disk and machine compatibility rules +129/-0

Share GKE disk and machine compatibility rules

• Introduces conservative Hyperdisk-only series classification, attach-error parsing, and cause/action text. Determines disk types only for GCE volumes with dynamic-provisioning provenance and a matching StorageClass.

pkg/gkedisk/gkedisk.go

Bug fix (1) +120 / -3

detect_scheduling.goExplain the real cause behind attach backoffs +120/-3

Explain the real cause behind attach backoffs

• Correlates attach events by current pod and volume, substituting an informative failure for a later backoff echo. Produces cause and action from the Compute Engine error or, when only echoes remain, from cached node and provisioned-volume data.

internal/k8s/detect_scheduling.go

Tests (5) +685 / -3

pd_placement_test.goTest cached audit findings and denied access +72/-0

Test cached audit findings and denied access

• Exercises the cache-to-audit path for a PD-backed StatefulSet and verifies that a reader without cluster-resource grants receives no placement finding.

internal/audit/pd_placement_test.go

scope_test.goVerify placement grant resolution +37/-3

Verify placement grant resolution

• Tests that placement resources receive cluster-scoped grant checks and that missing StorageClass access suppresses evidence. Adjusts existing scope tests for the additional grant lookups.

internal/audit/scope_test.go

detect_postbind_attach_test.goCover attach backoffs and event correlation +185/-0

Cover attach backoffs and event correlation

• Tests recovery of older informative failures, cache-derived causes, predecessor UID rejection, and per-volume matching. Also verifies that compatible nodes and unrelated timeouts receive no disk-compatibility cause.

internal/k8s/detect_postbind_attach_test.go

disk_compat_test.goTest placement constraints and finding severity +281/-0

Test placement constraints and finding severity

• Covers owner attribution, template changes, node selectors, affinity, taints, volume zones, and warning-versus-danger findings. Confirms that inapplicable evidence produces no evaluation.

pkg/audit/disk_compat_test.go

gkedisk_test.goTest disk classification and attach explanations +110/-0

Test disk classification and attach explanations

• Tests machine-series classification, error and backoff recognition, disk-type provenance, and wording for fully versus partially PD-compatible series.

pkg/gkedisk/gkedisk_test.go

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Old replicas hide disk placement risks 🐞 Bug ≡ Correctness

Description

pdPlacementOwner groups ReplicaSet pods under a Deployment, but pdPlacementTemplates then
evaluates the Deployment's current template instead of the ReplicaSet template that creates those
pods. During a rollout, an older ReplicaSet can still create pods with different node constraints,
so a newly pinned Deployment can incorrectly appear safe while those replicas remain eligible for
Hyperdisk-only nodes.

Code

pkg/audit/disk_compat.go[R190-192]

+	for _, d := range input.Deployments {
+		out[ResourceKey("apps", "Deployment", d.Namespace, d.Name)] = &d.Spec.Template.Spec
+	}

Evidence

The owner lookup maps a ReplicaSet-controlled pod to its Deployment, and that key selects only the
Deployment template. The repository also uses the UID-matched ReplicaSet template when it needs the
spec that creates replacement pods.

pkg/audit/disk_compat.go[104-110]
pkg/audit/disk_compat.go[188-199]
pkg/audit/disk_compat.go[213-221]
internal/issues/provider_pod_template.go[54-66]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Deployment-level attribution currently substitutes the Deployment's current template for the template of the ReplicaSet that creates each pod.
## Fix Focus Areas
- pkg/audit/disk_compat.go[88-110]
- pkg/audit/disk_compat.go[188-224]
## Recommended Fix
Keep findings attributed to the Deployment, but evaluate each pod's replacement eligibility using its UID-matched owning ReplicaSet template. Cover a rollout where old and new ReplicaSets have different node constraints.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Volume changes trigger misleading audits 🐞 Bug ≡ Correctness

Description

checkPDVolumeHyperdiskPlacement takes PVCs and bound-volume constraints from the observed pod but
takes node constraints from its owner's current template. If a Deployment template removes a
Persistent Disk claim while an older pod still mounts it, the check can warn that replacement pods
will fail to attach a volume they no longer use; adding a claim can likewise leave a replacement
risk unevaluated.

Code

pkg/audit/disk_compat.go[R73-77]

+		for _, vol := range pod.Spec.Volumes {
+			if vol.PersistentVolumeClaim == nil {
+				continue
+			}
+			key := pod.Namespace + "/" + vol.PersistentVolumeClaim.ClaimName

Evidence

Claims and PVs are collected exclusively from the observed pod before the check substitutes the
owner template for scheduling. The collector's evidence is keyed by bound claims, so that
substitution does not update the volume set being assessed.

pkg/audit/disk_compat.go[71-85]
pkg/audit/disk_compat.go[102-110]
internal/audit/pd_placement.go[62-74]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The placement check combines an observed pod's PVCs with its owner's current scheduling template, which can describe a different set of volumes.
## Fix Focus Areas
- pkg/audit/disk_compat.go[71-110]
- pkg/audit/disk_compat.go[188-199]
## Recommended Fix
Evaluate replacement placement against the claims in the template that creates the replacement pod, resolving their bound PV constraints where available. Keep the observed pod's claims for the separate already-placed assessment, and test a template that adds or removes a claim.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Attach echoes blame the wrong disk ✓ Resolved

Description

attachFailureDiagnosis handles a backoff echo by returning an incompatibility for any Persistent
Disk mounted by the pod, without checking which volume emitted the echo. On a Hyperdisk-only node
with multiple PVCs, an echo for a Hyperdisk volume can therefore receive a Persistent Disk cause and
action even when the event has no informative predecessor.

Code

internal/k8s/detect_scheduling.go[R1614-1616]

+	for _, diskType := range podGCEDiskTypes(cache, pod) {
+		if gkedisk.IsPersistentDisk(diskType) {
+			return gkedisk.IncompatibilityCause(diskType, machineType), gkedisk.IncompatibilityAction(diskType)

Evidence

Event selection parses the echoed volume only for informative-event lookup, then calls diagnosis
without that identity. The fallback iterates disk types for every PVC on the pod and returns upon
finding any Persistent Disk.

internal/k8s/detect_scheduling.go[1444-1453]
internal/k8s/detect_scheduling.go[1587-1595]
internal/k8s/detect_scheduling.go[1607-1616]
internal/k8s/detect_scheduling.go[1649-1664]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The backoff fallback diagnoses every mounted GCE disk rather than the volume named by the selected attach event.
## Fix Focus Areas
- internal/k8s/detect_scheduling.go[1444-1453]
- internal/k8s/detect_scheduling.go[1603-1619]
- internal/k8s/detect_scheduling.go[1633-1666]
## Recommended Fix
Pass the selected event's volume identity into the fallback and resolve only that volume's PVC and provisioned disk type. Leave cause and action empty when the echoed volume cannot be identified; test an echo for a Hyperdisk volume alongside a separate Persistent Disk PVC.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (1)

4. Hyperdisk attach errors get the wrong cause and fix ✓ Resolved

Description

ParseIncompatibleAttach accepts any disk type, but IncompatibilityCause and
IncompatibilityAction assume it is a Persistent Disk. A Hyperdisk volume the machine can't take
(e.g. hyperdisk-extreme disk type cannot be used by n4-standard-4 machine type) gets the cause
"hyperdisk-extreme volumes can't attach to n4 nodes: the N4 machine series supports only Hyperdisk",
which contradicts itself, and a Hyperdisk volume on a PD-only series (e.g. n2) is told to "move the
data to a Hyperdisk StorageClass", which is the type that already fails.

Code

pkg/gkedisk/gkedisk.go[R117-122]

+func IncompatibilityCause(diskType, machineType string) string {
+	if HyperdiskOnly(machineType) {
+		return fmt.Sprintf("%s volumes can't attach to %s nodes: the %s machine series supports only Hyperdisk.",
+			diskType, machineType, strings.ToUpper(MachineSeries(machineType)))
+	}
+	return fmt.Sprintf("Compute Engine doesn't allow %s disks on %s machines, so the volume never attaches.", diskType, machineType)

Evidence

attachFailureDiagnosis passes whatever disk type the Compute Engine error contains straight into
IncompatibilityCause/IncompatibilityAction (detect_scheduling.go 1604-1605) and never checks
IsPersistentDisk. IncompatibilityCause picks the "supports only Hyperdisk" wording based only on
HyperdiskOnly(machineType), and IncompatibilityAction always recommends a Hyperdisk
StorageClass. Both strings end up in the issues and diagnose Cause/Action fields.

pkg/gkedisk/gkedisk.go[64-75]
pkg/gkedisk/gkedisk.go[116-129]
internal/k8s/detect_scheduling.go[1603-1606]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`IncompatibilityCause` uses the "supports only Hyperdisk" wording, and `IncompatibilityAction` recommends moving to Hyperdisk, even when the rejected disk type is itself a Hyperdisk type. A parsed `hyperdisk-*` error therefore gets a cause that contradicts itself and a fix that points at the disk type that already failed.
## Fix Focus Areas
- pkg/gkedisk/gkedisk.go[116-129]
- internal/k8s/detect_scheduling.go[1603-1606]
## Recommended Fix
Use the Hyperdisk-only wording only when `IsPersistentDisk(diskType) && HyperdiskOnly(machineType)`; otherwise use the generic "Compute Engine doesn't allow X disks on Y machines" cause. Make `IncompatibilityAction` suggest a Hyperdisk StorageClass only for PD disk types. For Hyperdisk disk types, suggest pinning to a machine series that supports that Hyperdisk type, or a StorageClass of a supported type. Add tests for `hyperdisk-extreme` on n4 and `hyperdisk-balanced` on n2.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can tweak Display settings with a live preview to see your comment before it ships

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[nadaverell]

Live verification (post-merge, on 7abd60120)

Repro: a temporary tainted 1-node n4d-standard-2 pool on a GKE 1.35 cluster, plus a StatefulSet pinned to it with a 4Gi premium-rwo (pd-ssd) claim. The attach failed as expected. GKE emitted both events:

• ControllerPublish not permitted on node … due to backoff condition. In this run it appeared only during the first few seconds.
• googleapi: Error 400: pd-ssd disk type cannot be used by n4d-standard-2 machine type., badRequest. This became the newest event and repeated on every retry. The live error code is InvalidArgument, not Internal; the parser doesn't depend on it.

The captured live objects and events were run through DetectPostBindProblems in three event states. States 2 and 3 used the real event text with adjusted timestamps.

Event state	Before (9ae4cc6bb)	After (7abd60120)
1. Real error newest (as observed)	—	Real error shown · cause + action ✓
2. Backoff echo newest (outage shape)	Echo only · no cause/action	Real error shown · cause + action ✓
3. Real error expired, echo only	Echo only · no cause/action	Echo shown · cause derived from node + PV/StorageClass ✓

Cause: pd-ssd volumes can't attach to n4d-standard-2 nodes: the N4D machine series supports only Hyperdisk.

Audit (pdVolumeHyperdiskOnlyNode) on real cluster snapshots:

• Repro cluster: 1 finding out of 666, StatefulSet k1-test/pd-on-n4d, danger. The cluster's other PD workloads were not flagged; the taint keeps them off the node.
• Production-shape cluster (e2 + n4d nodes, Redis on pd-ssd pinned to e2): 0 findings out of 605.
• The same snapshot with the Redis nodeSelector removed, which is the outage configuration: 2 warnings, one for redis-master and one for redis-replicas, each naming the n4d node and its claims. This ran on the pre-7462bfdba branch head.

The test pool, namespace and disk were deleted afterwards.