Repository navigation
fix(gitops): surface why a Degraded Argo app is actually Degraded - #1624
Merged
nadaverell merged 7 commits intoSep 13, 2026
Merged
Conversation
âĶactually looks The backend already computes a concrete cause (crashloop/OOM/image-pull/ unschedulable/etc.) per Degraded or Missing managed resource, but the frontend only rendered it on non-headline issues inside an expanded disclosure â the common single-issue case never showed it. Also wire the same cause data into the resource tree so an unhealthy node's hover tooltip explains why, not just that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
âĶo doesn't Argo's own aggregate app health can roll up to Degraded from a managed resource whose kind never writes a per-resource health.status into status.resources[] at all â ClusterSecretStore, ClusterExternalSecret, and other CRDs outside Argo's built-in Lua health checks are common offenders. Every existing detector (operation failure, per-resource Degraded/Missing, the tree-summary fallback) comes up empty in that case, leaving the operator with an unexplained Degraded badge and no way to tell which of N managed resources is responsible. Fall back to the managed resource with the loudest recent Warning event as the best available attribution â verified live against external-secrets-packages on kiac-man, where the real cause (a ClusterSecretStore failing to authenticate) was sitting in an event with 17 occurrences, invisible anywhere in the Issues band before this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PR Summary by QodoSurface root causes for Degraded Argo applications
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
âĶ-count Warning event as real signal Two real bugs: - refKey matched tree nodes to their Issue cause by kind+namespace+name only. A core resource and a same-plural CRD in a different group (Knative Service vs core Service, CNPG vs CAPI Cluster â CLAUDE.md's own documented collision examples) sharing a namespace+name would collide, showing one resource's degradation cause/tooltip on a completely unrelated node. Added group to the key. - degradedResourceFromEvents required ev.Count > 0 to consider a Warning event, using 0 as its "nothing found yet" sentinel. A real, first-occurrence Warning commonly reports Count == 0 on modern clusters (events.k8s.io/v1 only sets a count once an event has repeated into a series) â exactly the kind of signal this fallback exists to catch was being silently treated as no signal at all. Sentinel is now -1, so a genuine zero-count event is correctly recognized as found while still losing to a higher-count sibling. Also removed a test comment that only restated the assertion right below it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Argo CD 3.x stops persisting per-resource health in the Application by default, which is why status.resources[] carried nothing to point at. The fallback now asks the issues engine about each managed resource first and only then falls back to the loudest Warning event.
Contributor
|
Thanks for the fix, the headline cause and the tree tooltip were real gaps. One correction on the diagnosis: Argo CD does have health checks for the external-secrets.io kinds, so it did evaluate that ClusterSecretStore as Degraded. It just never showed up in I pushed two small commits on your branch: the comments now describe that cause, the fallback checks what Radar already knows about each managed resource before using your Warning-event pick, and it only calls something Degraded when there's a strong signal. Otherwise it shows the resource as a lead ("may be why") next to the degraded count. CI is green, merging. A proper fix for the Argo CD 3 case is coming in a follow-up. Your PR is what surfaced it. |
The attribution tier now scans every managed resource and prefers a
critical issues-engine finding over a warning-tier one that merely comes
first. Only a critical finding is stated as Degraded; a warning-only
finding or a Warning-event pick is a warning-severity lead ('may be why')
that leaves the degraded-resources summary visible. The tier runs only
for an app deploying to this cluster, and an informational Running or
drift row no longer suppresses it.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
â Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3136940. Configure here.
âĶry their detail as Cause Only a failed sync operation or a critical per-resource Issue accounts for degraded managed resources; StuckDriftLoop and ManualDrift are sync signals and no longer suppress the attribution or the tree summary. The 'may be why' leads put their detail in Cause so the tree tooltip and the headline's second line show it.
The issues-engine step is what the per-resource health follow-up does properly, so it goes there instead. What stays: the Warning-event lead (warning-tier, 'PossibleCause', detail in Cause for the tree tooltip), gated to in-cluster apps and not counted as explaining the app's health.
nadaverell
approved these changes
Sep 13, 2026
nadaverell
added a commit
that referenced
this pull request
Sep 13, 2026
âĶ1743) Argo CD 3.0 stopped persisting per-resource health into the Application object by default (`status.resourceHealthSource: appTree`). Every Radar reader of `status.resources[].health` went blind on such installs: no per-resource Issues, no cause bridge to the issues engine, `Unknown` tree nodes for non-topology kinds, no Degraded/Missing categories in the Resources table. The app-level badge still said Degraded with nothing under it. #1624 surfaced this. ## What changes **One resolved health per managed resource, with provenance.** A tree node's `health` now carries `healthSource` (`controller` for the GitOps CR's own verdict, `radar` for Radar's read) plus `healthReason` / `healthMessage` / `healthSeverity`. Every surface reads that one value: node colour (previously from Radar's topology status even when Argo said Degraded), the Resources-table category, per-resource Issues, and the summary counts. `argoResourceChanges` takes health from the tree instead of re-reading the CR, and the tree and insights endpoints share one `resolveGitOpsTree` path so a node can't differ between the two responses. The wire field `health` stays a string, so `@skyhook-io/k8s-ui` consumers are unaffected. **Radar's diagnostics fill the gap, labelled as Radar's.** In appTree mode, for an app that is not Healthy and deploys to this cluster, the host overlays the issues engine's existing classification (crash loops, image pulls, failed Ready conditions; already transient-filtered and owner-collapsed) onto declared nodes. Problems only, never `Healthy`; controller health is final; a node whose health was only Radar's topology read gives way to the engine's more specific finding. The composition is memoized per user and namespace set for 5s (`gitopsIssuesMemo`) so a page load's two requests and the 2s polls share one pass. **Severity follows the evidence.** An engine finding keeps its own reason and severity as a per-resource Issue. A Radar topology read with no classified reason is a warning-tier observation, not a verdict, and neither it nor the "N managed resources are degraded" count appears on an app Argo calls Healthy. The count and the Warning-event lead from #1624 only show when no per-resource finding already names a resource. **The UI says so.** A notice above the Issues band: *"Argo CD 3 no longer records health for each resource on the application. Rows marked "Found by Radar" are problems Radar detected by looking at the resources itself."* with a "Why?" link to the docs. When Radar's own look found nothing to mark, the notice says that and names the way out (`controller.resource.health.persist: "true"` in `argocd-cmd-params-cm`). Rows and declared nodes carry a "Found by Radar" marker with the engine's message in the tooltip. **Remote-destination apps read nothing local.** An Application deploying to another cluster used to be matched against same-named local objects (topology, cache, drift, events, cause bridge). It now carries only what the CR says, with a notice: *"This application deploys to a different cluster, so its resources aren't visible from here."* Standalone Radar, when the Cloud funnel is available, adds *"Connect both clusters to Radar Cloud to see this app's resources together."*, where "Radar Cloud" opens the existing funnel dialog. **Fixture.** `gitops-demo` installs Argo CD v3.5.2 by default (server-side apply; `ARGOCD_VERSION=v2.13.2` selects the older inline shape). New `argo3-health` app sourced from this repo (`ARGO3_FIXTURE_REVISION` to test a branch): a CRD Argo evaluates via a Lua health check installed into `argocd-cm`, one it ignores, and a crashlooping Deployment. `guestbook-remote` covers the remote-destination case. **Docs.** `docs/gitops.md` "Per-resource health"; one-line rule in CLAUDE.md; demo README matrix. The in-product "Why?" link targets `/features/gitops#per-resource-health` on radar-docs, added by skyhook-dev/radar-docs#94 â merge that before the release that carries this. Not in this PR: sourcing Argo's own verdicts from the argocd-server API when the integration is connected (fast-follow), and `Missing` derivation for declared-but-absent resources. ## Testing - `go test ./...` in both modules, `make tsc`, k8s-ui vitest (108). - Visual, on a fresh Argo CD 3.5.2 kind cluster: `argo3-health` (notice, engine-sourced headline, "Found by Radar" on the three declared nodes and in the Resources table, none on generated ReplicaSet/Pod; light and dark) and `guestbook-remote` (notice; the Cloud sentence opening the funnel dialog with `RADAR_CLOUD_FUNNEL=on`). - Argo CD 2.x parity (controller health wins, no marker, no notice) is pinned by unit tests; not re-run live. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes how GitOps detail surfaces health and issues for Argo CD 3 and remote appsâoperator-facing diagnosis pathsâbut rules limit false positives (no synthetic Healthy, skip remote/healthy-app overlay) and coverage is heavy on unit tests. > > **Overview** > Adds **per-resource health with provenance** for Argo CD 3âs default (`appTree`), where the Application CR no longer carries `status.resources[].health`. Tree and insights now share **`resolveGitOpsTree`**, which can **`overlayRadarHealth`** from the issues engine (problems only, memoized ~5s per user/namespaces) so tree nodes, Resources table, and Issues stay aligned via `healthSource` (`controller` | `radar`) plus reason/message/severity. > > **Insights and tree behavior** change accordingly: `argoResourceChanges` resolves health from the tree (not a second CR read); per-resource Issues respect engine severity and suppress contradictions when Argo calls the app Healthy; remote-destination Applications skip local topology/drift/events/cause overlay. The tree builder tags **`healthMode`** / **`remoteDestination`** and stops attributing local objects to spoke destinations. > > **UI** adds a banner (`GitOpsHealthSourceNotice`), **âFound by Radarâ** markers on tree/Changes rows, docs links, and standalone **Radar Cloud** hint for remote apps (`openCloudFunnel`). **Demo/docs** default to Argo **v3.5.2** with `argo3-health` and `guestbook-remote` fixtures. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7c58e43. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An Argo Application's aggregate health can go Degraded for reasons that are genuinely hard to trace back to a specific managed resource today. Two real gaps, found by live-testing against real clusters:
1. The backend already computes a cause, the frontend didn't show it in the common case.
buildIssuesattaches a concreteCause(crashloop / OOM / image-pull / unschedulable / etc., bridged from the cluster-wide issues engine) to a per-resource Degraded/Missing issue â but the UI only renderedcauseon non-headline issues inside an expanded disclosure. The single-issue case (the common one â one degraded resource) never showed it. Fixed to render on the headline row unconditionally. Also wired the same cause data into the resource tree, so an unhealthy node's hover tooltip explains why, not just that.2. Some Degraded apps generate zero Issues at all.
Argo CD 3.0 stopped persisting per-resource health in the Application object by default (
status.resourceHealthSource: appTree). Argo still evaluates each resource and rolls the result up into the app's health, butstatus.resources[].healthstays empty for every kind. Every existing detector (operation failure, per-resource Degraded/Missing, the tree-summary fallback) reads that field, so a Degraded app can produce zero Issues and the operator can't tell which of N managed resources is responsible.Added a fallback tier for that case: when the app is Degraded and nothing else explained it, Radar first checks what it already knows about each managed resource (crashloops, failed Ready conditions and so on), and otherwise attributes the badge to the managed resource with the loudest recent Warning event. Verified live: an app with 4 managed resources, none with individual health, where the actual cause (a
ClusterSecretStorefailing to authenticate, 17 occurrences of the same Warning event) was invisible in the Issues band before and now surfaces directly.Test plan
go test ./pkg/gitops/insights/...(new tests: cause-on-headline rendering via k8s-ui,TestBuildIssues_DegradedAppFallsBackToLoudestResourceEvent)npx vitest run(k8s-ui gitops components)make tscðĪ Generated with Claude Code
Note
Medium Risk
Adds event-based attribution and shared in-cluster destination logic on the insights path; behavior is gated and warning-tier, but it can surface misleading leads if events are stale.
Overview
Improves why a Degraded Argo app looks broken when per-resource health is missing (Argo CD 3.x) or when causes were already computed but hidden in the UI.
Backend: When an Application is Degraded and nothing else explains it (no critical per-resource issue, no failed op), insights adds a warning-tier
PossibleCauseissue by scanning managed resources for the loudest recent Warning event (includingCount == 0). That path runs only for in-cluster destinations; it does not replace real explanations, so the DegradedResources tree summary can still appear.IsInClusterDestinationis moved topkg/gitopsand reused from the diff gate.UI: Issue headline rows show
causeunder the main line (not only when expanded). The resource tree graph takes the same insights issues and shows cause tooltips on unhealthy/degraded nodes via a stable ref key (group/kind/namespace/name).Reviewed by Cursor Bugbot for commit 5ed6f24. Bugbot is set up for automated code reviews on this repo. Configure here.