Per-resource health with provenance when Argo CD doesn't persist it

CLAUDE.md

@@ -217,6 +217,7 @@ WebSocket pod exec: `internal/server/exec.go` — xterm.js terminal, container/s
   - **Nested navigation**: `classifyGitOpsKind` tags nodes with `data.gitopsTool` + `data.gitopsKind`. Portal nodes route to child detail pages; lineage breadcrumb (`?from=kind|ns|name`) enables back navigation.
   - **Severity vocabulary**: `critical` (0, red) → `alert` (1, orange) → `warning` (2, amber) → `info` (3, blue). Adding a new severity requires updating both Go `severityRank` and TS union in `gitops-insights.ts`.
   - **Single-cluster limitation**: Application↔resource edges only render when controller + workloads are in same cluster (ArgoCD hub-spoke deployments won't show connections).
+  - **Per-resource health**: read a tree node's resolved `health` + `healthSource` (`controller` | `radar`), never `status.resources[].health` directly — Argo CD 3 doesn't persist it, and `overlayRadarHealth` (`internal/server/gitops_handlers.go`) fills the gap from the issues engine. See [docs/gitops.md](docs/gitops.md#per-resource-health)
   - **Per-resource drift**: computed from `kubectl.kubernetes.io/last-applied-configuration` annotation. SSA/Helm-installed resources lack this; SSA fallback tracked in [#601](https://github.com/skyhook-io/radar/issues/601).
 
 ### Timeline + resource relationships

docs/gitops.md

@@ -62,13 +62,24 @@ The Issues band at the top of the detail page surfaces six classes of problems:
 - **Stuck-drift loop** — when sync succeeded but the app is *still* OutOfSync with auto-sync on and a recent reconcile, something is mutating resources after each apply. The Issue calls out likely culprits (mutating webhook, sibling controller, schema migration)
 - **Manual drift without auto-sync** — drift exists but auto-sync is disabled. The Issue tells you "nothing will reconcile until you click Sync" so you stop waiting
 - **Argo Application conditions** — `ComparisonError` (verify repo creds), `OrphanedResourceWarning`, `InvalidSpecError`, etc. extracted into typed-severity Issues
-- **Per-resource health** — Degraded / Missing children get a critical Issue each, deduped against any operation failure that already named the same resource (no triplicate rendering)
+- **Per-resource health** — Degraded / Missing children get a critical Issue each, deduped against any operation failure that already named the same resource (no triplicate rendering). See [Per-resource health](#per-resource-health) for where that health comes from
 - **Pending deletion** (lifecycle) — see [Lifecycle awareness](#lifecycle-awareness) below
 
 **Structured remediation** — when the diagnosis pipeline recognizes a fixable failure (e.g. Argo operation error "namespace X not found"), the Issue carries a primary-blue action button that performs the fix in one click. Duplicate per-resource Missing issues + SyncError condition rows are then suppressed so the user sees one clear "create the namespace and retry" path instead of three.
 
 While an operation is running, the page polls every 2s; otherwise on-demand.
 
+### Per-resource health
+
+The health chip on each managed resource (tree node, Resources table, per-resource Issues) comes from one of two places, and Radar tells you which:
+
+- **Argo CD's own verdict.** Argo evaluates every managed resource it has a health check for (built-in checks for core kinds and many CRDs, plus your `resource.customizations`) and rolls the results up into the Application's health. Up to Argo CD 2.x this per-resource result was also written into the Application object (`status.resources[].health`), and that is what Radar reads.
+- **Radar's own read.** Argo CD 3.0 changed the default: the controller no longer persists per-resource health into the Application (`status.resourceHealthSource: appTree`, controlled by `controller.resource.health.persist` in `argocd-cmd-params-cm`). The app-level health still rolls up, but every managed resource in the object is left without one. When Radar sees that, it reads the live resources itself through its issues engine (crash loops, image pulls, failed `Ready` conditions and so on) and reports what it finds. Those rows carry a **Radar** marker, and a notice above the Issues band says that the problems listed are Radar's findings. Radar only reports problems this way; it never marks a resource Healthy on Argo's behalf, and a resource it has nothing on is shown without a health value.
+
+Two things follow from that. A resource Argo has no health check for (so Argo ignores it) can still show a Radar finding if it is unhealthy: that is Radar's read, not a change in Argo's verdict, and the marker says so. And an Application that deploys to another cluster gets neither: Radar can only read the cluster it is connected to, so it derives nothing and says why.
+
+If you want Argo's per-resource verdicts back in the object, set `controller.resource.health.persist: "true"` in `argocd-cmd-params-cm` and restart the application controller. Radar switches back to Argo's values automatically.
+
 ## Lifecycle awareness
 
 When a GitOps resource is being deleted (`metadata.deletionTimestamp` set), its Sync and Health values are leftovers from the last reconcile *before* deletion was triggered. Showing them as if they're current produces contradictory state ("Syncing · Progressing · Terminating") that misleads operators into trying to fix routine sync problems on a resource the cluster is actively tearing down.

internal/server/gitops_handlers.go

@@ -146,15 +146,87 @@ func (s *Server) handleGitOpsTree(w http.ResponseWriter, r *http.Request) {
 		})
 		return
 	}
-	tree, _, err := s.buildGitOpsTree(r.Context(), req)
+	tree, _, _, err := s.resolveGitOpsTree(r, req)
 	if err != nil {
 		s.writeGitOpsBuildError(w, req, err)
 		return
 	}
-	tree = s.filterGitOpsTreeForUser(r, req, tree)
 	s.writeJSON(w, tree)
 }
 
+// resolveGitOpsTree is the one path both detail endpoints take from a parsed
+// request to a servable tree: build, overlay Radar's own health where the
+// controller's isn't in the CR, drop what the user may not see, then count
+// what's left. The tree and insights endpoints are separate requests; sharing
+// this keeps a node's health identical in both responses. The returned
+// resolver already carries this request's access gate so insights can reuse
+// it (and its once-per-request issue-engine composition) instead of building
+// a second one.
+func (s *Server) resolveGitOpsTree(r *http.Request, req *gitopsRequest) (*gitopstree.ResourceTree, *unstructured.Unstructured, *insightsResolver, error) {
+	tree, root, err := s.buildGitOpsTree(r.Context(), req)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	canAccess := func(group, kind, namespace, name string) bool {
+		return s.canAccessGitOpsRef(r, req, group, kind, namespace, name, false)
+	}
+	resolver := newInsightsResolver(r.Context(), req.Cache, req.AllowedNamespaces, canAccess)
+	memoKey := gitopsIssuesMemoKey(auth.UserFromContext(r.Context()), req.AllowedNamespaces)
+	resolver.composed = func() ([]issues.Issue, []issues.Issue) {
+		return s.gitopsIssuesMemo.load(memoKey, resolver.composeIssues)
+	}
+	overlayRadarHealth(tree, root, resolver.ResourceProblems)
+	tree = s.filterGitOpsTreeForUser(r, req, tree)
+	tree.Summary = gitopstree.Summarize(tree.Nodes)
+	return tree, root, resolver, nil
+}
+
+// overlayRadarHealth fills per-resource health from Radar's issues engine
+// when the Argo Application doesn't carry the controller's own
+// (status.resourceHealthSource=appTree, the Argo CD 3 default). It only
+// ever adds problems: a node the engine has nothing on keeps no health, so
+// nothing here claims a resource is Healthy on Argo's behalf. Skipped for a
+// remote destination (Radar's engine reads the local cluster, which could
+// hold an unrelated same-named object) and for an app Argo calls Healthy
+// (the engine may still hold a warning; contradicting the controller's
+// verdict on a node, and composing the cluster's issues on every poll of a
+// healthy app, are both worse than leaving it). A node whose health is
+// Radar's own topology read (Progressing at 1/2 replicas, say) is still a
+// candidate: the engine's classified finding is the more specific Radar
+// answer and replaces it. Controller-sourced health is final. problems is
+// the resolver's ResourceProblems, which applies the request's per-user
+// access gate.
+func overlayRadarHealth(tree *gitopstree.ResourceTree, root *unstructured.Unstructured, problems func(group, kind, namespace, name string) []gitopsinsights.ResourceProblem) {
+	if tree == nil || root == nil || problems == nil {
+		return
+	}
+	if tree.HealthMode != gitopstree.HealthModeAppTree || tree.RemoteDestination {
+		return
+	}
+	if appHealth, _, _ := unstructured.NestedString(root.Object, "status", "health", "status"); appHealth == "Healthy" {
+		return
+	}
+	for i := range tree.Nodes {
+		n := &tree.Nodes[i]
+		if n.Role != gitopstree.RoleDeclared {
+			continue
+		}
+		if n.Health != "" && n.HealthSource != gitopstree.HealthSourceRadar {
+			continue
+		}
+		problem, ok := gitopsinsights.WorstResourceProblem(problems(n.Ref.Group, n.Ref.Kind, n.Ref.Namespace, n.Ref.Name))
+		if !ok {
+			continue
+		}
+		n.Health = "Degraded"
+		n.HealthSource = gitopstree.HealthSourceRadar
+		n.HealthReason = problem.Reason
+		n.HealthMessage = problem.Message
+		n.HealthSeverity = problem.Severity
+		n.TopologyStatus = "unhealthy"
+	}
+}
+
 func (s *Server) handleGitOpsInsights(w http.ResponseWriter, r *http.Request) {
 	if !s.requireConnected(w) {
 		return
@@ -169,17 +241,17 @@ func (s *Server) handleGitOpsInsights(w http.ResponseWriter, r *http.Request) {
 		})
 		return
 	}
-	tree, root, err := s.buildGitOpsTree(r.Context(), req)
+	tree, root, resolver, err := s.resolveGitOpsTree(r, req)
 	if err != nil {
 		s.writeGitOpsBuildError(w, req, err)
 		return
 	}
-	tree = s.filterGitOpsTreeForUser(r, req, tree)
-	canAccess := func(group, kind, namespace, name string) bool {
-		return s.canAccessGitOpsRef(r, req, group, kind, namespace, name, false)
+	// A remote-destination app's resources aren't in this cluster; the
+	// insights builder won't ask for them, so don't fan out GETs for
+	// same-named local objects either.
+	if !tree.RemoteDestination {
+		resolver.prefetchLive(gitopsinsights.ManagedResourceRows(root), gitopsinsights.OperationPhase(root))
 	}
-	resolver := newInsightsResolver(r.Context(), req.Cache, req.AllowedNamespaces, canAccess)
-	resolver.prefetchLive(gitopsinsights.ManagedResourceRows(root), gitopsinsights.OperationPhase(root))
 	insight := gitopsinsights.Build(root, tree, resolver)
 	insight.Warnings = appendWarnings(insight.Warnings, tree.Warnings...)
 	insight = s.filterGitOpsInsightForUser(r, req, insight)
@@ -509,7 +581,7 @@ func (s *Server) filterGitOpsTreeForUser(r *http.Request, req *gitopsRequest, tr
 	if keep[tree.Root.ID] {
 		out.Root = tree.Root
 	}
-	out.Summary = summarizeGitOpsTree(filteredNodes)
+	out.Summary = gitopstree.Summarize(filteredNodes)
 	out.Warnings = appendWarnings(append([]string{}, tree.Warnings...), "Some managed resources are hidden by RBAC.")
 	return &out
 }
@@ -603,27 +675,6 @@ func appendWarnings(existing []string, warnings ...string) []string {
 	return existing
 }
 
-func summarizeGitOpsTree(nodes []gitopstree.Node) gitopstree.Summary {
-	var s gitopstree.Summary
-	for _, n := range nodes {
-		switch n.Role {
-		case gitopstree.RoleDeclared:
-			s.Declared++
-		case gitopstree.RoleGenerated:
-			s.Generated++
-		case gitopstree.RoleGroup:
-			s.Grouped += n.Count
-		}
-		if n.Health == "Degraded" || n.Health == "Missing" {
-			s.Degraded++
-		}
-		if n.Sync == "OutOfSync" {
-			s.OutOfSync++
-		}
-	}
-	return s
-}
-
 func (s *Server) canAccessGitOpsRef(r *http.Request, req *gitopsRequest, group, kind, namespace, name string, root bool) bool {
 	if auth.UserFromContext(r.Context()) == nil {
 		return true
@@ -679,6 +730,9 @@ type insightsResolver struct {
 	composeOnce     sync.Once
 	composedFlat    []issues.Issue
 	composedGrouped []issues.Issue
+	// composed, when set by the host, supplies the composition (memoized
+	// across requests); nil composes inline, which tests and other hosts use.
+	composed func() ([]issues.Issue, []issues.Issue)
 
 	// livePrefetched flips GetLive into map-only mode: prefetchLive resolved
 	// (or deliberately skipped) every ref the Changes builder will ask for,
@@ -855,14 +909,11 @@ func (r *insightsResolver) ResourceProblems(group, kind, namespace, name string)
 		return nil
 	}
 	r.composeOnce.Do(func() {
-		r.composedFlat = issues.Compose(issues.NewCacheProvider(), issues.Filters{
-			Namespaces: r.allowedNamespaces,
-			Limit:      issues.NoLimit,
-			CanReadRelated: func(ref issues.Ref) bool {
-				return r.canAccess != nil && r.canAccess(ref.Group, ref.Kind, ref.Namespace, ref.Name)
-			},
-		})
-		r.composedGrouped = issues.GroupIssues(r.composedFlat)
+		if r.composed != nil {
+			r.composedFlat, r.composedGrouped = r.composed()
+			return
+		}
+		r.composedFlat, r.composedGrouped = r.composeIssues()
 	})
 	related := issues.RelatedIssuesFrom(r.composedFlat, r.composedGrouped, issues.RelatedIssueOptions{
 		CanReadRelated: func(ref issues.Ref) bool {
@@ -966,6 +1017,19 @@ func (r *insightsResolver) RecentEvents(group, kind, namespace, name string) []g
 	return out
 }
 
+// composeIssues runs the cluster-wide issues engine for this request's
+// namespaces, redacting related refs the caller can't read.
+func (r *insightsResolver) composeIssues() ([]issues.Issue, []issues.Issue) {
+	flat := issues.Compose(issues.NewCacheProvider(), issues.Filters{
+		Namespaces: r.allowedNamespaces,
+		Limit:      issues.NoLimit,
+		CanReadRelated: func(ref issues.Ref) bool {
+			return r.canAccess != nil && r.canAccess(ref.Group, ref.Kind, ref.Namespace, ref.Name)
+		},
+	})
+	return flat, issues.GroupIssues(flat)
+}
+
 // FinalizerOwnerStatus implements gitopsinsights.Resolver.
 //
 // Looks up the controller responsible for a finalizer (via the static

internal/server/gitops_health_overlay_test.go

@@ -0,0 +1,97 @@
+package server
+
+import (
+	"testing"
+
+	gitopsinsights "github.com/skyhook-io/radar/pkg/gitops/insights"
+	gitopstree "github.com/skyhook-io/radar/pkg/gitops/tree"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+)
+
+func overlayApp(health string) *unstructured.Unstructured {
+	return &unstructured.Unstructured{Object: map[string]any{
+		"apiVersion": "argoproj.io/v1alpha1",
+		"kind":       "Application",
+		"metadata":   map[string]any{"name": "app", "namespace": "argocd"},
+		"status":     map[string]any{"health": map[string]any{"status": health}, "resourceHealthSource": "appTree"},
+	}}
+}
+
+func overlayTree(mode gitopstree.HealthMode, remote bool) *gitopstree.ResourceTree {
+	return &gitopstree.ResourceTree{
+		HealthMode:        mode,
+		RemoteDestination: remote,
+		Nodes: []gitopstree.Node{
+			{Role: gitopstree.RoleRoot, Ref: gitopstree.ResourceRef{Group: "argoproj.io", Kind: "Application", Namespace: "argocd", Name: "app"}, Health: "Degraded"},
+			{Role: gitopstree.RoleDeclared, Ref: gitopstree.ResourceRef{Group: "external-secrets.io", Kind: "ClusterSecretStore", Name: "platform"}},
+			{Role: gitopstree.RoleDeclared, Ref: gitopstree.ResourceRef{Group: "apps", Kind: "Deployment", Namespace: "prod", Name: "web"}, Health: "Progressing", HealthSource: gitopstree.HealthSourceRadar, TopologyStatus: "degraded"},
+			{Role: gitopstree.RoleDeclared, Ref: gitopstree.ResourceRef{Group: "apps", Kind: "Deployment", Namespace: "prod", Name: "api"}, Health: "Healthy", HealthSource: gitopstree.HealthSourceController, TopologyStatus: "healthy"},
+			{Role: gitopstree.RoleDeclared, Ref: gitopstree.ResourceRef{Kind: "ConfigMap", Namespace: "prod", Name: "vars"}},
+			{Role: gitopstree.RoleGenerated, Ref: gitopstree.ResourceRef{Kind: "Pod", Namespace: "prod", Name: "web-1"}},
+		},
+	}
+}
+
+func fakeProblems(calls *[]string) func(group, kind, namespace, name string) []gitopsinsights.ResourceProblem {
+	return func(group, kind, namespace, name string) []gitopsinsights.ResourceProblem {
+		*calls = append(*calls, kind+"/"+name)
+		switch name {
+		case "platform":
+			return []gitopsinsights.ResourceProblem{
+				{Reason: "Ready: InvalidProviderConfig", Message: "no route to host", Category: "condition_false", Severity: "warning"},
+			}
+		case "web", "api":
+			return []gitopsinsights.ResourceProblem{{Reason: "CrashLoopBackOff", Message: "back-off", Severity: "critical"}}
+		}
+		return nil
+	}
+}
+
+func TestOverlayRadarHealth_FillsOnlyEmptyDeclaredNodes(t *testing.T) {
+	tree := overlayTree(gitopstree.HealthModeAppTree, false)
+	var calls []string
+	overlayRadarHealth(tree, overlayApp("Degraded"), fakeProblems(&calls))
+
+	css := tree.Nodes[1]
+	if css.Health != "Degraded" || css.HealthSource != gitopstree.HealthSourceRadar || css.HealthReason != "Ready: InvalidProviderConfig" || css.HealthMessage != "no route to host" || css.HealthSeverity != "warning" || css.TopologyStatus != "unhealthy" {
+		t.Errorf("ClusterSecretStore should carry the engine's finding as Radar-sourced Degraded, got %+v", css)
+	}
+	if dep := tree.Nodes[2]; dep.Health != "Degraded" || dep.HealthReason != "CrashLoopBackOff" || dep.HealthSeverity != "critical" || dep.TopologyStatus != "unhealthy" {
+		t.Errorf("Radar's topology read (Progressing) must give way to the engine's classified finding, got %+v", dep)
+	}
+	if api := tree.Nodes[3]; api.Health != "Healthy" || api.HealthSource != gitopstree.HealthSourceController || api.HealthReason != "" {
+		t.Errorf("controller-sourced health is final; the engine must not overwrite it, got %+v", api)
+	}
+	if cm := tree.Nodes[4]; cm.Health != "" || cm.HealthSource != "" {
+		t.Errorf("a node the engine has nothing on must stay without health, got %+v", cm)
+	}
+	for _, c := range calls {
+		if c == "Pod/web-1" || c == "Deployment/api" {
+			t.Errorf("generated nodes and controller-assessed nodes must not be looked up; saw %q", c)
+		}
+	}
+}
+
+func TestOverlayRadarHealth_SkipsInlineModeRemoteDestinationAndHealthyApp(t *testing.T) {
+	cases := []struct {
+		name string
+		tree *gitopstree.ResourceTree
+		app  *unstructured.Unstructured
+	}{
+		{"inline mode", overlayTree(gitopstree.HealthModeInline, false), overlayApp("Degraded")},
+		{"remote destination", overlayTree(gitopstree.HealthModeAppTree, true), overlayApp("Degraded")},
+		{"healthy app", overlayTree(gitopstree.HealthModeAppTree, false), overlayApp("Healthy")},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			var calls []string
+			overlayRadarHealth(tc.tree, tc.app, fakeProblems(&calls))
+			if len(calls) != 0 {
+				t.Errorf("engine must not be consulted, got calls %v", calls)
+			}
+			if tc.tree.Nodes[1].Health != "" {
+				t.Errorf("no health may be overlaid, got %+v", tc.tree.Nodes[1])
+			}
+		})
+	}
+}