Repository navigation
Commit fbfe986
authored
Detect custom resources stuck deleting on an operator's finalizer (#2053)
VictoriaMetrics' operator was uninstalled while its VMAgent and VMSingle
objects still carried `apps.victoriametrics.com/finalizer`. They stayed
deleting for weeks, kept workloads running, and left the Helm release
stuck uninstalling. Radar's terminating detector covered built-in
resources but missed these custom resources.
This adds stuck-deletion findings for watched custom resources and
explains their finalizers. The primary advice is to repair or restore
the controller. Manual removal appears separately, only for reviewed
catalog entries and only under an explicit intentional-uninstall or
permanent-scale-down condition.
### Behavior and read rules
- Scan existing watched CRD stores; start no new watches. After 10
minutes deleting, emit a warning. After 30 minutes, controller-backed
findings become critical only with evidence of an installed controller
that is down or not fully healthy. Healthy, unidentified, unrecognized
or unreadable controllers stay warning. Protection and
garbage-collection finalizers retain their documented severity rules.
- Use #2071's subject/evidence authorization: ordinary namespaced
subjects require namespace access; Secrets require the additional Secret
read; cluster-scoped subjects require exact-kind list. Controller
inventory follows the same evidence granularity: namespace access for
namespaced reads and exact-kind list for cluster-wide reads. Incomplete
inventory cannot establish a missing/down controller or enable cleanup.
A synchronized namespace still produces findings when another
namespace's informer has not synchronized.
- Keep #2071's per-resource coverage envelope and withheld count. A user
allowed to get a cluster-scoped object but not list its kind sees that a
finding was withheld, without its payload.
- Shared issue composition feeds REST/MCP issues, per-resource findings,
diagnose, GitOps and CNPG. The home Active Issues panel uses
`/api/issues`. AI resource-list and search summaries include the
composed findings and incur their scan cost; `/api/dashboard`, MCP
`get_dashboard` and `cluster://health` exclude the dynamic scan.
- Fold deletion conditions on the same subject into its termination row.
Time the issue from the deletion request; the UI labels it “since
deletion.”
### Controller recognition and boundaries
`pkg/finalizers/catalog.go` owns pure finalizer identity, workload/image
matching and per-controller cleanup policy. It recognizes Argo CD, Flux,
VictoriaMetrics and Karpenter. Detection is generic across watched CRDs;
controller-specific diagnosis is catalog-driven. New recognition
defaults to cleanup disabled. The catalog replaces the exported
`insights.FinalizerOwner` / `insights.ResolveFinalizerOwner` names;
current supported Hub code has no call sites for them. The integration
guide documents where to add entries and the positive/negative fixtures
required.
Recognition uses workload names/labels, pod-template labels and
published image repository paths. Registry hosts, tags and digests are
ignored; mirrors must preserve the vendor repository path as trailing
segments; proxy-cache prefixes are supported. Flux Helm requires the
published image path because unrelated Helm controllers share its name
and labels. A matching name/label with an unrecognized image leaves
identity unconfirmed and blocks cleanup instead of counting as an absent
controller. Role predicates distinguish Argo application controllers
from other components sharing their image, and VictoriaMetrics operators
from config reloaders. Name-only guesses for uncataloged finalizers
remain candidates: they cannot establish ownership, change severity or
unlock cleanup.
The inventory checks Deployments, StatefulSets, DaemonSets and
recognized pods elsewhere. All copies contribute to health; a healthy
copy cannot hide a failing one. Reuse observations by resolved
controller within each caller's composition. Health summaries live in
`pkg/health` and are shared with GitOps. Argo CD and Karpenter can run
outside the cluster and never opt into cleanup; Karpenter also releases
infrastructure.
### Conditional cleanup
No match is not proof of an uninstall. The finding says “No recognized
controller found,” stays warning and leads with restoration. Only Flux
and VictoriaMetrics catalog entries opt into `conditional_cleanup` after
a complete negative inventory or permanent-scale-down candidate.
The additive field contains the user-confirmed condition, agent
guidance, caution and dry-run commands. Each JSON Patch tests the object
UID and exact finalizer at its index before removing one finalizer.
Re-read after each removal. The Issues UI keeps cleanup collapsed
beneath Next step; it performs no mutation. Removal releases only this
object’s finalizer. Older VM operators also finalize managed Kubernetes
objects, so deleting the parent can leave children stuck and Pods
running; restoring the operator after the parent is gone may not clean
them up. Repair/restore advice says the controller can resume cleanup,
without guaranteeing completion. MCP guidance prohibits presenting
removal as the next step before the user confirms the condition.
Mixed finalizers retain the existing warning policy when one controller
is healthy, while preserving repair, restore and check advice for every
other finalizer. Unknown controllers, weak candidates, incomplete
access, off-cluster installations and protection finalizers get no
removal commands.
### Validation
- Documentation-only follow-up at `244e84263`: simplified all three
changed Markdown files by 61 lines, corrected recovery wording and the
conventional-namespace GitOps lookup description, and retained safety
rules and extension points. Scope, relative links and whitespace
checked; behavior is unchanged.
- Full frontend/embed/Go build and affected Issues/MCP suites at
`7fa46cb63`; conditional-cleanup UI tests passed. The broader root/`pkg`
suites and 2,023 web / 4,504 k8s-ui tests (one existing skipped) remain
evidence for unchanged paths from the preceding integration checks.
- Regression coverage includes deletion thresholds, exact resource
identity, per-namespace synchronization,
namespace/Secret/cluster-subject authorization, inventory completeness,
renamed/image-role recognition, multi-copy health, unknown/mixed
finalizers, cleanup opt-in, condition folding and UID/index guards.
- **Actual VM lifecycle on existing EKS, no kind:** Helm-renamed,
namespace-scoped VictoriaMetrics operator v0.68.1 with real
VMAgent/VMSingle workloads and operator-image reloaders. At `b7cd15ff5`,
intentional operator uninstall left three real CRs deleting with healthy
application/reloader Pods. No findings before ten minutes; afterward
warning, restore-first advice and conditional cleanup. Reloaders were
not mistaken for the operator. REST/MCP agreed; generated server dry run
succeeded; wrong UID/finalizer guards failed unchanged; one reviewed
apply deleted only its owned parent CR.
- Restoring the operator deleted the two parents left intact and their
main workloads. The bypassed parent’s seven built-in dependents stayed
stuck, and all three auto-generated VMServiceScrapes remained deleting
under this older operator. Those leftovers were inspected and cleaned
separately with UID/finalizer guards. The operator release, owned
namespace and all 26 created empty CRDs were removed.
- At `72b521d1c`, the actual healthy `vmop` operator was recognized by
image on all three naturally stuck scrape CRs: warning, no cleanup,
REST/MCP agreement. After intentional scale-to-zero, the corrected
dependent-finalizer caution and collapsed/expanded UI were verified;
generated previews and guarded applies removed only those owned scrape
CRs. The `7fa46cb63` follow-up changes recovery wording/documentation
and test copy only; unchanged behavior reuses those live runs.
- **Shared-mode EKS at `9e60c1573`:** stand-in fixtures with proxy
authentication and three RBAC identities cover healthy, crashloop,
image-pull failure, Deployment/DaemonSet without Pods, scaled-zero and
unrecognized states; namespace-only callers get neither cleanup nor
hidden controller names. REST/MCP agree; get-only cluster coverage
reports a withheld finding; unreadable Secret evidence loses
names/keys/onset. UI captures cover collapsed cleanup, restricted
omission and withheld messaging. Those fixtures were 10–30 minutes
deleting; the 30-minute critical ramp has tests and earlier live no-pod
evidence at `07101583e`.
**Limits:** object-level conditional cleanup is not a complete VM
installation uninstall. The default Radar chart’s deployed
ServiceAccount/Hub permissions, a live AI confirmation conversation,
large-cluster performance and persistent/cloud-data cleanup remain
unverified. v0.75.0 was installed and healthy during setup, but the
intentional uninstall/recovery failure reproduction used v0.68.1. The
live run used Helm for the operator and kubectl for the VM CRs; it did
not reproduce the original application chart’s Helm-release bookkeeping.
Earlier kind evidence is historical; this EKS run created no kind
cluster. [Review
packet](https://radar-pr-2053-review.nadav168665.chatgpt.site)
(owner-private).1 parent 072ddd1 commit fbfe986
37 files changed
Lines changed: 3526 additions & 373 deletions
File tree
- docs
- internal
- issues
- k8s
- mcp
- server
- summarycontext
- packages/k8s-ui/src/components/issues
- pkg
- finalizers
- gitops/insights
- health
- issuesapi
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
91 | 114 | | |
92 | 115 | | |
93 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
491 | 491 | | |
492 | 492 | | |
493 | 493 | | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
494 | 543 | | |
495 | 544 | | |
496 | 545 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
141 | 142 | | |
142 | 143 | | |
143 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
144 | 149 | | |
145 | 150 | | |
146 | 151 | | |
| |||
155 | 160 | | |
156 | 161 | | |
157 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
158 | 166 | | |
159 | 167 | | |
160 | 168 | | |
161 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
162 | 173 | | |
163 | 174 | | |
164 | 175 | | |
165 | 176 | | |
166 | 177 | | |
167 | 178 | | |
168 | 179 | | |
| 180 | + | |
169 | 181 | | |
170 | 182 | | |
171 | 183 | | |
| |||
0 commit comments