Repository navigation
Detect custom resources stuck deleting on an operator's finalizer - #2053
Merged
Merged
Conversation
After an operator was uninstalled in a real cleanup, its VMAgent and VMSingle objects kept their finalizer for weeks, holding a Helm uninstall open while the workloads kept running on a 20Gi disk. Radar's terminating detection only covered typed kinds. Already-watched, synced custom resources with a deletion timestamp now raise the same issue (warning after 10 minutes, critical after 30, onset at the deletion time), naming the finalizers and how long deletion has been waiting. Where the finalizer maps to a known controller (the GitOps catalog plus VictoriaMetrics, or an operator workload matched by name or app label), the issue says whether running controller pods were observed, only within namespaces the caller can read and the cache covers; otherwise it says the controller is unknown. The advice starts with the controller's logs and permissions, and only for an intentionally removed controller shows an exact patch_resource JSON Patch that tests the UID and finalizer value before removing one finalizer, warning that its cleanup is skipped.
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
PR Summary by QodoDetect custom resources stuck deleting on operator finalizers
AI Description
Diagram
High-Level Assessment
Files changed (9)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
…alizer patches The custom-resource termination scan is its own provider method used only by the issues path, so dashboard surfaces no longer list cluster-scoped custom resources a caller can't read. In-use protection finalizers (*.k8s.io, kubernetes.io) get an explanation of what still references the object instead of a removal patch, Karpenter is in the controller catalog, and no patch is offered while the controller is running. Controller health uses the GitOps page's summarizer, now shared, so a crashlooping controller is reported as such. Severity stays at warning while a matched controller is running, a Deleting condition for the same object folds into the issue, the advice adds the kubectl form, and namespace-restricted callers scan custom resources once rather than once per namespace.
A controller with some ready pods (mid-rollout, or HA with a ready leader) may be doing the cleanup, so the removal patch is no longer offered then; severity is only capped when every pod is healthy. Karpenter's provider finalizers (karpenter.k8s.aws, karpenter.azure.com) map to Karpenter. Generic condition rows on deleting objects stay visible as before, the GitOps controller line no longer reads degraded during a rollout, the shared helpers are documented (including that a finalizer owner may have no namespace), and tests pin that both issues handlers pass the list check.
A stuck custom resource now gets a ready-made finalizer-removal patch only when Radar identified the finalizer's controller (catalog or workload match) and saw it stopped after a complete cluster-wide workload search. Unknown controllers, which today include Crossplane, Cluster API, Config Connector, ACK and Strimzi, get advice to find and check the controller, no patch, and stay at warning instead of going critical. Karpenter and Argo CD can run off-cluster (EKS Auto Mode, AKS node auto-provisioning, Amazon EKS Capabilities), so finding none of their pods is reported as unconfirmed, not gone. Karpenter's finalizers drain nodes and terminate instances, so it never gets a removal patch; a self-hosted Karpenter seen down is critical with advice to restore it. x-k8s.io finalizers named for in-use or protection (Kueue resource-in-use) are treated as in-use guards. Flux ImageUpdateAutomation now resolves to image-automation-controller.
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
A controller with a workload but no ready pod (crashlooping, image pull failure, pending, or unable to create pods) was treated as stopped and offered the finalizer-removal patch. It will finish the deletion once fixed, so removing its finalizer skips cleanup it still intends to do. Only an absent controller now gets the patch: a complete search found no matching workload or catalog pod, or its Deployment or StatefulSet is scaled to 0. An installed controller with no ready pod stays critical after the threshold with advice to fix it and no patch. Absence also needs DaemonSets searched, catalog workloads match on their pod-template labels, and catalog pods outside the matched workloads are checked before a scaled-down copy counts as absent. The in-use guard advice no longer lists examples that only fit some guards.
Terminating rows carried the generic creation/after-healthy timing chip, so a row whose clock counts from deletionTimestamp read "30m since creation". Rows with deletion-based timing now read "since deletion" and "pending since deletion was requested", for typed and custom resources alike. The termination_stuck catalog description no longer claims the owning controller is unhealthy; Radar also raises it when the controller is healthy or unknown.
…ller absent A stopped in-cluster Argo CD or Karpenter copy no longer counts as absent: a managed install outside the cluster may still own the finalizer, so it is reported as unconfirmed with no removal patch. Catalog controller pods are searched cluster-wide rather than only in the conventional namespace. A healthy install beside a crashlooping or partly ready one no longer caps the issue at warning.
…atalog pod The cluster-wide catalog search could find a ready pod elsewhere and mark the controller healthy while a matched workload was crashlooping, capping the stuck deletion at warning.
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
…ditional cleanup Catalog controllers (Argo CD, Flux, VictoriaMetrics, Karpenter) are now also recognized by the repository path of their published image, on pods and on Deployment/StatefulSet/DaemonSet pod templates, ignoring registry host, tag and digest. A renamed install (Helm nameOverride) or a mirror that keeps the path is recognized; a renamed Deployment with replicas but no pods reads as down. Argo CD's shared image counts only when the container runs argocd-application-controller. A search that recognizes nothing is no longer called absent. It reads "No recognized controller found" at warning, and a catalog controller scaled to 0 lands in the same bucket. The next step is to restore the controller; the finalizer removal moves out of the action into a new additive conditional_cleanup field (condition, agent guidance, caution, and the UID/index-tested dry-run patch per finalizer). The MCP issues description tells agents to propose it only after the user confirms the condition, and the Issues UI shows it as a collapsed section labeled with the condition. A first-DNS-label name match (cluster.example.io vs rabbitmq-cluster-operator) now names a candidate only: it never decides the controller's state, the severity, or a cleanup. Recognized workloads and catalog pods outside them are always merged, so a healthy workload no longer hides a crashlooping copy elsewhere. Orphan-only finalizers say orphaning keeps the dependents. With the Pod cache unavailable, a catalog owner is still named and only its state is reported as unknown.
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
nadaverell
added a commit
that referenced
this pull request
Oct 12, 2026
…claims, Released PVs) (#2055) In a real cleanup every piece of unused storage was found by hand: - about 311Gi of Loki and Mimir disks left behind after their apps were deleted - about 45Gi of 2.7-year-old PVCs in `default` - an 8Gi Postgres PVC with no workload - a PVC that survived its Helm uninstall - two PVCs Pending for about three years on a WaitForFirstConsumer storage class, which the issue detector deliberately ignores - Released PVs stuck in a `VolumeFailedDelete` retry loop Cluster Audit now reports these as **snapshot** Efficiency checks (Medium on the existing Checks and MCP `get_cluster_audit` surfaces): | Check | Reports | |---|---| | `pvcNoConsumer` | Bound PVCs older than a day with **no consumer Radar can observe**: Pods, Deployments, ReplicaSets, StatefulSets (including zero replicas and leftover ordinals after a scale-down), DaemonSets, Jobs, CronJobs, generic ephemeral volumes, and Pending PVC clones (a clone protects its source only while it is Pending). On Kubernetes 1.37+, the claim's `Unused` condition is used when present (below). Shows size, storage class, age, bound PV and reclaim policy. | | `pvcLongPending` | PVCs Pending for over 24h with no observed consumer, WaitForFirstConsumer included. Shown as "Pending since creation"; a claim never returns to Pending. A Pending claim that *has* a consumer is already an outage issue, so it isn't double-reported as waste. | | `releasedPV` | Released PVs: those kept by a `Retain` policy (after a 24h grace when release time is known, or flagged with "release time unknown" and PV age when it is not), and those whose deletion hasn't completed after an hour of known release duration. A recent `VolumeFailedDelete` warning establishes deletion failure immediately, even without a phase-transition timestamp. Delete PVs without that timestamp or a recent warning are not evaluated. **Failed** PVs (an in-tree delete or recycle failed after the claim was deleted) are reported immediately, with the latest current `VolumeFailedDelete`/`VolumeFailedRecycle` warning or else the PV's `status.message`. | **Not waste by construction:** - **Owned PVCs are skipped.** A PVC with a controller owner reference, such as a hibernated CloudNativePG cluster's volumes, a stopped KubeVirt VM's disks, or workflow `volumeClaimTemplates`, follows its owner's lifecycle, so these checks leave it to the owner. - **The check says "no consumer observed"**, never "unused" or "safe to delete". The caveat about consumers Radar can't see (CRDs mounting volumes intermittently, snapshot operations) lives in the check description (shown once per check) and the remediation (returned with every MCP finding), not on every row. Rows state facts only: size, class, age, bound PV, reclaim policy, and the `Unused` time or binding-mode evidence where present. - **Remediation says how to reclaim.** No consumer: if nothing needs the data, delete the claim; if its PV is Retain, reclaim that too. Retain PVs: deleting the PV keeps its disk; for a dynamically provisioned volume set the reclaim policy to Delete, otherwise delete the PV and then the disk in your cloud provider. Both keep the "confirm nothing needs the data" step first. **Kubernetes 1.37 `Unused` condition** (PVC protection controller, beta, on by default): - `Unused=False` (a running Pod uses the claim) counts as a consumer, even if Radar's Pod inventory missed it. - With `Unused=True`, the 24h floor counts from when the claim became unused instead of from its creation, so a claim whose workload was deleted minutes ago isn't flagged. The finding adds "Kubernetes reports it unused by any Pod since <time> (5d ago)". It reports the condition rather than promising a minimum idle time: upstream notes the time can be off in either direction because of controller delays. Right after the feature is enabled (e.g. an upgrade to 1.37), every unused claim's timestamp is the enable time, so long-unused claims are held back for a day and then reported with that shorter age. - No condition or `Unknown` keeps the creation-age behavior; absence never implies unused. `pvcLongPending` doesn't use it. **Deleting claims and volumes → `stuckTerminating`:** - PVCs and PVs are now `stuckTerminating` subjects, with the same 5m/30m ramp as other kinds. Deleting PVCs were already skipped by the storage checks; deleting PVs are now skipped by `releasedPV` too, so a deleting Bound PV no longer counts as a pass and a deleting Released PV isn't reported twice. - For `kubernetes.io/pvc-protection` and `kubernetes.io/pv-protection`, the finding explains what holds the deletion and what happens when it's released (the claim is deleted and the volume follows its reclaim policy), suggests a backup if the deletion was unintended, and says not to remove the finalizer. It never tells the user to stop Pods or delete the claim: a deletion can't be cancelled, and completing it can delete the data. - The shared `stuckTerminating` remediation (shown in Checks and MCP) now exempts storage finalizers from its "clear finalizers as a last resort" advice. **Coverage:** - Inputs are gated on cache readiness. PVs and StorageClasses are cluster-scoped and checked against the caller's exact grants per kind; PVCs, workload templates, and events follow namespace visibility. - `missingInputs` keeps its existing meaning: something was not evaluated. A storage input is recorded only when its absence left a claim or volume unevaluated, so unreadable evidence never looks like a clean pass: - an unreadable PVC inventory (`persistentvolumeclaims`) - unknown consumers (`pvc-consumers`) - a PV inventory the caller may read but Radar couldn't list (`persistentvolumes`) - a Released Delete volume still in its 1h grace period whose outcome depends on events Radar can't see (`pv-deletion-events`) - a Released Delete volume with a nil or zero phase-transition timestamp and no recent deletion warning: release duration is unknown, so PV creation age cannot establish a stalled deletion or a pass (`pv-release-time`) - Gaps that still let a finding fire are stated in the finding instead: "reclaim policy not visible", "StorageClass binding mode not visible", "Deletion warning events not fully visible". Radar's "Some checks could not run" banner and Hub's "checks skipped" copy stay accurate, so no consumer change is needed. - Without a PV grant, Released PVs aren't subjects of that caller's scan, so namespace-restricted users don't get a permanent missing input. - **Failed PVs never pass and are never skipped.** The PV controller sets `Failed` only when reclaiming failed, so the phase alone establishes the finding. Events only add detail; incomplete coverage adds the "Deletion warning events not fully visible" caveat, never `pv-deletion-events`. The controller emits the warning once, at the transition, and events expire (1h by default), so `status.message` is the fallback. - **What `releasedPV` counts:** Bound PVs pass, Released and Failed PVs are judged, and Available/Pending PVs are not counted. They have no claim to have outlived, so counting them as passes would only inflate the pass rate (for example, a local-static-provisioner pool). No Radar-kept usage history, cost column or delete action; those are follow-ups. `pkg/audit` changes are additive. **Testing** - Table tests cover Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, CronJobs, generic ephemeral volumes, and Pending clones through both reference fields (including owned, young, and deleting destinations). Cross-namespace `dataSourceRef` references protect only the source in the referenced namespace. Bound clones, non-core references, and VolumeSnapshot references do not protect a source. Tests also cover StatefulSet replicas 0 and leftover ordinals, controller-owned (CNPG) claims, the 24h age floor at a fixed clock, WaitForFirstConsumer with and without consumers, and Retain vs Delete with and without events. Nil and zero release timestamps cover Delete skips, immediate recent-warning findings, and Retain findings with unknown release time and PV age. They also cover missing inputs only for skipped subjects (otherwise the caveat is in the finding), PV inventories that are unavailable vs. outside the caller's grant, and unreadable namespaces suppressing findings. - `Unused` condition: False with no observed consumer and with incomplete inventory, True 2h ago on a 30-day-old claim, the exact 24h boundary, future and zero timestamps, True 10 days ago with the evidence sentence, zero-replica StatefulSet and Pod consumers, absent and `Unknown` conditions, and Pending claims ignoring it. - Audit counts, MCP audit scope (including `pv-release-time` appearing only with a PV grant), and the `ChecksView` input labels. - Failed PVs: a Delete PV with a current warning (event text and age), with only `status.message`, with partial or unreadable events (caveat, evaluated, not passed, no missing input), with a warning older than the Failed transition (ignored), Recycle with `VolumeFailedRecycle`, and a policy switched to Retain after the failure. Counts: Bound passes; Failed is evaluated but not passed; Available and Pending are not counted. `internal/audit` collects events for Failed PVs. - `make test` and `make tsc` pass. The original change was checked live on a kind cluster (an unconsumed bound PVC and a Released Retain PV through REST and MCP). **Live verification** (kind, Kubernetes v1.36.1 so `lastPhaseTransitionTime` is populated; real binary in local and proxy-auth modes; REST + MCP + Checks UI) - **Earlier round** (independent live tester, head `3dc5f6a3`): - The prior proof re-run passed. - Unreadable StorageClasses produce the "binding mode not visible" caveat and no missing input. - A namespace user without a PV grant gets no `persistentvolumes` missing input. - A real skip (no Pod list) is listed as `pvc-consumers`. - A Released Delete PV past its grace period is flagged, with the "not fully visible" caveat under incomplete events. - A CSI `VolumeFailedDelete` on a Released PV is reported as "deletion is failing". - Warnings from before the current release are ignored. - The Checks view shows the "could not run" banner only on real skips. - MCP `get_cluster_audit` matches REST. - The WaitForFirstConsumer, default-class and class-not-found messages (readable StorageClasses) are covered by unit tests only. - **That round found this bug:** a hostPath Delete PV whose in-tree deletion failed went to `Failed` with a real `VolumeFailedDelete` event, and was counted as a `releasedPV` **pass**. - **After this fix** (head `4be95ee8`: the binary was built from `3dc5f6a3` plus the uncommitted fix, whose diff is identical to the `4be95ee8` commit; same repro: hostPath Delete PV at `/var/...`, claim deleted, PV `Failed` with `status.message` set and one event in `default`): - Admin view, `releasedPV` evaluated 2, passed 1 (the Bound PV passes, the Failed PV is flagged): `Failed PV: reclaiming the volume failed after its claim was deleted (reclaim policy Delete). Latest reclaim warning (2m ago): host_path deleter only supports /tmp/.+ but received provided /var/fix2055-delete-fail. Capacity 1Mi; storage class ""; former claim fix2055-f3/claim-delete-fail; Failed for 2m.` MCP shows the same text. - A proxy-auth user with a PV grant who sees only the fixture namespace, so the event in `default` is hidden: the same finding, carrying the `Status message: host_path deleter only …` text plus "Deletion warning events not fully visible". There is no `pv-deletion-events` missing input, and REST and MCP match. - The Checks UI renders the finding under "Released or Failed persistent volume". - Documentation commits and the README check-count test followed those live checks. The release-time uncertainty and Pending-clone scenarios have unit and MCP-handler coverage; their live verification remains pending. **Live verification on GKE 1.37.1** (nonprod control plane upgraded to 1.37.1; current head) - The controller set `Unused=False` on a claim with a running Pod and `Unused=True` (timestamped at Pod deletion) on two claims whose Pods were deleted. A `standard-rwo` (WaitForFirstConsumer) clone of one of them stayed Pending. - A snapshot of those live objects ran through the audit with creation and unused times shifted back to clear the 24h floor: | Variant | Result | |---|---| | As captured (claims minutes old) | No storage findings; only the pre-existing in-use claim evaluated, passing | | Created 10d ago, unused 5d | `orphan` flagged with the Unused time (5d ago); `clone-source` protected by the Pending clone; `in-use` passes; the abandoned Pending clone flagged as `pvcLongPending` | | Same, in-use Pod removed from the snapshot | `in-use` still passes on `Unused=False` | | Created 10d ago, unused since minutes ago | `orphan` and `clone-source` held back (not evaluated) | | Same as row 2, clone removed | `clone-source` flagged | - Deleting PVC/PV routing, protection-finalizer copy (including that it never advises completing the deletion) and the shared remediation's storage exception are covered in `pkg/audit` tests; all three fail on the previous head. - `make test`, `make tsc` and the audit/MCP suites pass on the current head, after merging main (which brought #2053). **Live check of stuck deletion** (GKE 1.37.1, head `9ea8f6871`): a PVC still mounted by a running Pod was deleted, then its bound PV. After 6 minutes both were reported by `stuckTerminating` as warnings: the PVC held by `kubernetes.io/pvc-protection`, the PV by `kubernetes.io/pv-protection` plus the two CSI finalizers, each with the protection explanation. `releasedPV` counted only the cluster's other (Bound) PV, so the deleting PV was neither passed nor double-reported. Deleting the namespace then removed the PVC, PV and disk. **Docs** - README states the Cluster Audit check count again (44, the size of `CheckRegistry`, which now includes the Persistent Disk placement check main added in #2069); `TestREADMECheckCountMatchesRegistry` fails when the two drift. - The storage review checks section in `docs/configuration.md` describes the consumer inventory, release-time uncertainty, thresholds, and evidence gaps; the MCP audit-access bullet links to it instead of repeating it. **Rollout and compatibility** - The MCP `get_cluster_audit` description now lists "stale PVCs/PVs" under Efficiency; the short wording keeps the tool catalog within `TestToolCatalogContextBudget` after main's growth. - The three checks are on by default at Medium, so posture counts (Medium findings, per-check pass rates) shift on upgrade for any cluster with unclaimed PVCs or Released/Failed PVs. Each check can be disabled in audit settings. - `releasedPV` counts Bound PVs as passes; Available and Pending PVs are not counted at all. - New `missingInputs` keys: `pvc-consumers`, `pv-deletion-events`, `pv-release-time`, plus `persistentvolumeclaims` / `persistentvolumes` when those inventories are unavailable. They appear only when a claim or volume went unevaluated, so Radar's "Some checks could not run" banner and Hub's "checks skipped" copy stay accurate. - `pkg/audit` changes are additive: new `CheckInput` fields and one exported predicate, `PVNeedsReclaimEvents`. Radar Hub imports `pkg/checks`, not `pkg/audit`, and keys checks by ID (titles are display-only), so an older Hub renders the new checks from the cluster's catalog. No new endpoint, so no feature flag is needed. **Known limits (product calls, not bugs)** - Findings within a check are ordered by severity, then kind, namespace and name (`pkg/checks.sortEffectiveFindings`), not by size, so a large disk can sit behind "View all". Size-first ordering needs a field on the shared `pkg/checks` finding that Hub would also carry; it is a follow-up. - `pvcLongPending` stays Medium like the other two checks. A Pending claim has no disk yet, but the detector can only emit High or Medium; Low is an org override in Hub. - StatefulSet scale-down leftovers are not flagged: every claim-template ordinal counts as consumed, so `data-kafka-4` after a 5→3 scale-down stays silent. - Consumers Radar can't see cause false positives, each worded "no consumer observed" with the CRD caveat. Examples: JupyterHub `claim-<user>` after culling, CI caches and backup targets mounted by short-lived Pods, KubeVirt VMs with direct PVC volumes, and snapshot sources or clone operations beyond the observed Pending PVC references. Ignored namespaces or disabling the check are the escape hatches. - PVC subjects follow namespace visibility, as other namespaced audit subjects do. Like those kinds, PVCs in a namespace that a namespace-scoped informer doesn't cover are not reported as a missing input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VictoriaMetrics' operator was uninstalled while its VMAgent and VMSingle objects still carried
apps.victoriametrics.com/finalizer. They stayed deleting for weeks, kept workloads running, and left the Helm release stuck uninstalling. Radar's terminating detector covered built-in resources but missed these custom resources.This adds stuck-deletion findings for watched custom resources and explains their finalizers. The primary advice is to repair or restore the controller. Manual removal appears separately, only for reviewed catalog entries and only under an explicit intentional-uninstall or permanent-scale-down condition.
Behavior and read rules
/api/issues. AI resource-list and search summaries include the composed findings and incur their scan cost;/api/dashboard, MCPget_dashboardandcluster://healthexclude the dynamic scan.Controller recognition and boundaries
pkg/finalizers/catalog.goowns pure finalizer identity, workload/image matching and per-controller cleanup policy. It recognizes Argo CD, Flux, VictoriaMetrics and Karpenter. Detection is generic across watched CRDs; controller-specific diagnosis is catalog-driven. New recognition defaults to cleanup disabled. The catalog replaces the exportedinsights.FinalizerOwner/insights.ResolveFinalizerOwnernames; current supported Hub code has no call sites for them. The integration guide documents where to add entries and the positive/negative fixtures required.Recognition uses workload names/labels, pod-template labels and published image repository paths. Registry hosts, tags and digests are ignored; mirrors must preserve the vendor repository path as trailing segments; proxy-cache prefixes are supported. Flux Helm requires the published image path because unrelated Helm controllers share its name and labels. A matching name/label with an unrecognized image leaves identity unconfirmed and blocks cleanup instead of counting as an absent controller. Role predicates distinguish Argo application controllers from other components sharing their image, and VictoriaMetrics operators from config reloaders. Name-only guesses for uncataloged finalizers remain candidates: they cannot establish ownership, change severity or unlock cleanup.
The inventory checks Deployments, StatefulSets, DaemonSets and recognized pods elsewhere. All copies contribute to health; a healthy copy cannot hide a failing one. Reuse observations by resolved controller within each caller's composition. Health summaries live in
pkg/healthand are shared with GitOps. Argo CD and Karpenter can run outside the cluster and never opt into cleanup; Karpenter also releases infrastructure.Conditional cleanup
No match is not proof of an uninstall. The finding says “No recognized controller found,” stays warning and leads with restoration. Only Flux and VictoriaMetrics catalog entries opt into
conditional_cleanupafter a complete negative inventory or permanent-scale-down candidate.The additive field contains the user-confirmed condition, agent guidance, caution and dry-run commands. Each JSON Patch tests the object UID and exact finalizer at its index before removing one finalizer. Re-read after each removal. The Issues UI keeps cleanup collapsed beneath Next step; it performs no mutation. Removal releases only this object’s finalizer. Older VM operators also finalize managed Kubernetes objects, so deleting the parent can leave children stuck and Pods running; restoring the operator after the parent is gone may not clean them up. Repair/restore advice says the controller can resume cleanup, without guaranteeing completion. MCP guidance prohibits presenting removal as the next step before the user confirms the condition.
Mixed finalizers retain the existing warning policy when one controller is healthy, while preserving repair, restore and check advice for every other finalizer. Unknown controllers, weak candidates, incomplete access, off-cluster installations and protection finalizers get no removal commands.
Validation
244e84263: simplified all three changed Markdown files by 61 lines, corrected recovery wording and the conventional-namespace GitOps lookup description, and retained safety rules and extension points. Scope, relative links and whitespace checked; behavior is unchanged.7fa46cb63; conditional-cleanup UI tests passed. The broader root/pkgsuites and 2,023 web / 4,504 k8s-ui tests (one existing skipped) remain evidence for unchanged paths from the preceding integration checks.b7cd15ff5, intentional operator uninstall left three real CRs deleting with healthy application/reloader Pods. No findings before ten minutes; afterward warning, restore-first advice and conditional cleanup. Reloaders were not mistaken for the operator. REST/MCP agreed; generated server dry run succeeded; wrong UID/finalizer guards failed unchanged; one reviewed apply deleted only its owned parent CR.72b521d1c, the actual healthyvmopoperator was recognized by image on all three naturally stuck scrape CRs: warning, no cleanup, REST/MCP agreement. After intentional scale-to-zero, the corrected dependent-finalizer caution and collapsed/expanded UI were verified; generated previews and guarded applies removed only those owned scrape CRs. The7fa46cb63follow-up changes recovery wording/documentation and test copy only; unchanged behavior reuses those live runs.9e60c1573: stand-in fixtures with proxy authentication and three RBAC identities cover healthy, crashloop, image-pull failure, Deployment/DaemonSet without Pods, scaled-zero and unrecognized states; namespace-only callers get neither cleanup nor hidden controller names. REST/MCP agree; get-only cluster coverage reports a withheld finding; unreadable Secret evidence loses names/keys/onset. UI captures cover collapsed cleanup, restricted omission and withheld messaging. Those fixtures were 10–30 minutes deleting; the 30-minute critical ramp has tests and earlier live no-pod evidence at07101583e.Limits: object-level conditional cleanup is not a complete VM installation uninstall. The default Radar chart’s deployed ServiceAccount/Hub permissions, a live AI confirmation conversation, large-cluster performance and persistent/cloud-data cleanup remain unverified. v0.75.0 was installed and healthy during setup, but the intentional uninstall/recovery failure reproduction used v0.68.1. The live run used Helm for the operator and kubectl for the VM CRs; it did not reproduce the original application chart’s Helm-release bookkeeping. Earlier kind evidence is historical; this EKS run created no kind cluster. Review packet (owner-private).