Skip to content

Stop three topology nodes reporting health nothing checked - #1668

Merged
nadaverell merged 1 commit into
mainfrom
fix/topology-status-false-claims
Sep 7, 2026
Merged

nadaverell merged 1 commit into
mainfrom
fix/topology-status-false-claims

Conversation

@nadaverell

@nadaverell nadaverell commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

A topology node's status is not just a dot in the graph. MCP builds its get_topology problem list from degraded and unhealthy nodes (internal/mcp/tools.go:1645), and GitOps derives managed-resource health from it (pkg/gitops/tree/graph.go:21). A green node is Radar answering "nothing is wrong" — to an operator scanning the graph, and to an AI asked whether the cluster is healthy.

Three places were answering that without checking. This continues #1645/#1649/#1659, which fixed the TypeScript accessors; pkg/topology computes health independently in Go and had drifted from them.

What changed

Gateway API routes were judged on Accepted alone. A gateway accepting a route says nothing about whether the backends it forwards to exist — so an HTTPRoute whose backendRef named a deleted Service reported healthy. That is precisely the failure an operator opens the graph to find, and it is the same defect #1645 fixed on the TS side. getRouteHealth now reads ResolvedRefs alongside Accepted.

A route reads healthy only when every parent reports both Accepted=True and ResolvedRefs=True. A parent that reports nothing — no conditions, or only ones observed against an earlier generation — is counted as unassessed and blocks both the healthy and the unhealthy verdict, so neither a green node nor a total failure is claimed while a gateway is still silent.

Two deliberate differences from the TS accessor, since this is otherwise meant to match it: the observedGeneration staleness check is an addition (speaksToCurrentSpec in resource-utils.ts:1953 only tests whether the relevant conditions exist), and a route whose parents all report nothing reads unknown here where TS reads Pending/degraded — claiming a fault we have not established seemed the wrong direction for this change. Both are candidates for reconciling in TS rather than divergences to keep.

HorizontalPodAutoscalers were hardcoded healthy while the builder read hpa.Status.CurrentReplicas three lines below. pkg/hpadiag already diagnoses HPAs for the detail drawer, so the node now uses it and the graph stops disagreeing with the drawer.

An autoscaler the controller has not written a condition for reads unknown rather than healthy: hpadiag.Analyze reports ok for a conditionless HPA, so absence of evidence would otherwise arrive as health. States the vocabulary calls intentionally idle — scaled to zero, disabled, pinned, stabilised, at minimum — map to neutral, matching the shipped TS mapping.

Otherwise the mapping only ever downgrades on negative evidence — unreachable metrics and inability to scale become unhealthy, capped/incomplete metrics degraded, everything else stays healthy. Notably scaling_up/scaling_down remain healthy: an HPA doing its job is not a problem, and since degraded nodes become MCP-reported problems, promoting routine autoscaling would manufacture false ones. This differs from hpaStateLevel in resource-utils-hpa.ts:68, which calls both scaling directions degraded — worth reconciling, but in both places deliberately rather than by divergence here.

Four Secret nodes are stubs synthesised from a reference — a TLS or CA secret named by a Traefik or HTTPProxy resource that the Secret listing never produced, either because it does not exist or because it was not listable. They were built green, so a misspelled certificateRef rendered as a healthy Secret. They now report unknown.

Verification

The HPA fix is verified end to end against a real controller failure rather than a hand-written fixture: a kind cluster has no metrics-server, so an HPA there genuinely gets ScalingActive=False / FailedGetResourceMetric from the HPA controller. With that in the cluster, the topology API returns status: "unhealthy" for the node, and get_topology over MCP returns the same — where both previously said healthy.

Route health is pinned by table-driven tests covering unresolved backendRefs, full acceptance, partial and total rejection, absent reports, a stale observedGeneration, and the mixed cases where one parent is current and healthy while another is stale or silent. I confirmed these actually fail against the previous algorithm rather than passing either way: for a route with Accepted=True, ResolvedRefs=False the old code returns healthy and the new code returns degraded.

go build and go test ./... clean in both the root and pkg modules.

Scope

Deliberately narrow. A survey found 22 hardcoded Status: StatusHealthy node constructions, but most are not defects worth changing:

  • Recolouring config objects (ConfigMap, listed Secrets, ServiceAccount, the synthetic Internet node) is not done. StatusNeutral means intentionally idle and paints a sky-blue outline (K8sResourceNode.tsx:275), so it would add emphasis to the least interesting nodes rather than remove a claim. That needs a deliberate "health not applicable" presentation instead.
  • NetworkPolicy and the spec-presence-implies-healthy sites (VirtualService, Istio Gateway, Traefik routes) are consistency gaps, not defects anyone is misled by.
  • Ingress loadBalancer and Service endpoints would be new health signals rather than removing a false one. Given that a wrong status propagates into MCP problems and GitOps degraded counts, a false alarm costs more than a missing one here, and Service semantics already live in internal/k8s/detect.go:1151.

Known and not addressed: TraefikService is marked unhealthy on zero backends in the resources view (builder.go:2274) but always healthy in the traffic view (:6472); Cilium and ClusterNetworkPolicy expose conditions the builder ignores; VPA conditions are readable and ConfigUnsupported currently renders green; and the generic condition helper (builder.go:7951) returns on the first Ready/Available/Succeeded it sees.

🤖 Generated with Claude Code


Note

Medium Risk
Changes how topology health is computed for routes, HPAs, and secrets, which can alter MCP-reported problems and GitOps degraded counts; logic is conservative but user-visible status may shift.

Overview
Topology nodes were marking healthy without checking real status, which flows into MCP problem lists and GitOps health. This PR tightens three cases in pkg/topology/builder.go.

Gateway API routes — getRouteHealth no longer treats Accepted=True as fully healthy. It also requires current ResolvedRefs (and ignores conditions on stale observedGeneration), counts silent parents as unassessed, and only returns healthy when every reporting parent accepts and resolves refs.

HPAs — HPA graph nodes call new hpaNodeHealth, which maps hpadiag.Analyze results into graph status (unhealthy/degraded for scale/metrics failures, neutral for intentional idle states, unknown when unreconciled). Routine scaling stays non-problematic so MCP does not get false alarms.

Referenced Secret stubs (Traefik/Contour TLS/CA refs not in the Secret list) switch from healthy to unknown so missing secrets are not shown as OK.

Adds node_health_test.go table tests for route and HPA health behavior.

Reviewed by Cursor Bugbot for commit edf6a40. Bugbot is set up for automated code reviews on this repo. Configure here.

@nadaverell
nadaverell requested a review from hisco as a code owner September 7, 2026 14:50
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Correct topology health for routes, HPAs, and referenced Secrets

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Evaluates Gateway routes using current Accepted and ResolvedRefs conditions.
• Maps HPA diagnostics into topology health without flagging routine scaling.
• Marks unverified referenced Secret stubs unknown and adds focused health tests.
Diagram

graph TD
  R["Gateway routes"] --> C{"Current conditions?"} --> T["Topology status"] --> O["MCP and GitOps"]
  H["HPA resources"] --> D["HPA diagnostics"] --> T
  S["Secret references"] --> U["Unknown status"] --> T
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Require explicit route confirmation
  • ➕ Avoids healthy status until both Accepted and ResolvedRefs are confirmed
  • ➕ Applies a stricter evidence standard consistently
  • ➖ Controllers omitting ResolvedRefs would leave otherwise functional routes unknown
  • ➖ Changes established compatibility behavior beyond the reported failures
2. Centralize cross-language health policy
  • ➕ Prevents future drift between Go topology and TypeScript accessors
  • ➕ Creates one authoritative resource-health contract
  • ➖ Requires a broader backend/frontend architectural refactor
  • ➖ Expands risk and review scope beyond this targeted correction

Recommendation: Keep the targeted negative-evidence approach for this fix: it detects confirmed failures without classifying routine scaling or controller omissions as problems. A follow-up should evaluate centralizing health policy across Go and TypeScript to prevent further drift.

Files changed (2) +233 / -13

Bug fix (1) +70 / -13
builder.goDerive honest health for routes, HPAs, and Secret stubs +70/-13

Derive honest health for routes, HPAs, and Secret stubs

• Uses hpadiag results for HPA node health, preserving healthy status for routine scaling while downgrading confirmed failures. Route health now evaluates current Accepted and ResolvedRefs conditions, and synthetic unverified Secret nodes become unknown instead of healthy.

pkg/topology/builder.go

Tests (1) +163 / -0
node_health_test.goCover route and HPA topology health decisions +163/-0

Cover route and HPA topology health decisions

• Adds focused tests for unresolved route references, rejected parents, stale generations, missing reports, HPA metric failures, scaling failures, healthy operation, routine scaling, and nil HPAs.

pkg/topology/node_health_test.go

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. New autoscalers appear healthy 🐞 Bug ≡ Correctness
Description
hpaNodeHealth maps hpadiag.StateOK to healthy through its default branch, although Analyze
also returns that state when the autoscaler has no controller conditions at all. A newly created or
never-reconciled autoscaler therefore gets a green topology node before the controller has confirmed
that it can read metrics or scale its target.
Code

pkg/topology/builder.go[R7904-7905]

+	default:
+		return StatusHealthy
Relevance

●●● Strong

Unverified HPA state conflicts with the PR’s declared-versus-confirmed health goal; analogous
false-green topology feedback was accepted.

PR-#1024

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Analyze initializes every non-nil autoscaler to StateOK, only adds reasons from status
conditions or replica signals, and returns StateOK when no reasons exist. The new topology mapper
handles only selected negative and stale states explicitly, so an object with an empty status
reaches the healthy default.

pkg/hpadiag/diagnosis.go[87-118]
pkg/hpadiag/diagnosis.go[234-262]
pkg/topology/builder.go[7892-7905]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Autoscalers without controller status are classified as healthy because the diagnostic analyzer's fallback state is treated as confirmed health.

## Issue Context
Distinguish a genuinely healthy diagnosis from an object whose status has not been populated. Empty conditions, and conditions that provide no conclusive controller verdict, should produce unknown rather than healthy.

## Fix Focus Areas
- pkg/topology/builder.go[7892-7906]
- pkg/topology/node_health_test.go[94-163]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Unverified route backends appear healthy 🐞 Bug ≡ Correctness
Description
getRouteHealth only counts ResolvedRefs=False, then returns healthy whenever every reporting
parent has Accepted=True, even if reference resolution is missing or Unknown. This occurs while
a controller is still resolving references or does not publish a conclusive resolution condition, so
nonexistent or inaccessible backends can still receive a green route node.
Code

pkg/topology/builder.go[R7954-7955]

+		if resolvedStatus == "False" {
+			unresolved++
Relevance

●● Moderate

The concern matches the PR’s false-green goal, but the added test deliberately preserves
accepted-only routes as healthy.

PR-#1024

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new code records all ResolvedRefs values but treats only False as negative, after which
accepted == reporting returns healthy. The adjacent comment itself states that route acceptance
says nothing about backend existence, while the added test explicitly encodes an accepted route with
no resolution report as healthy; Gateway API defines ResolvedRefs as the condition indicating
whether all route object references were resolved.

pkg/topology/builder.go[7934-7969]
pkg/topology/node_health_test.go[79-83]
🌐 Gateway API defines ResolvedRefs as the route condition indicating whether the controller resolved all object references, including missing backends.

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Route health becomes healthy from `Accepted=True` even when `ResolvedRefs` is absent or `Unknown`, although acceptance does not confirm that backend references resolve.

## Issue Context
A route parent should contribute healthy status only when its current conditions conclusively confirm both acceptance and reference resolution. Missing or unknown resolution should remain unknown or pending, while `ResolvedRefs=False` should continue downgrading health.

## Fix Focus Areas
- pkg/topology/builder.go[7922-7971]
- pkg/topology/node_health_test.go[38-84]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Scaled-to-zero workloads appear healthy 🐞 Bug ≡ Correctness
Description
hpaNodeHealth sends hpadiag.StateScaledToZero through its default healthy branch instead of
mapping it to the topology's neutral status. When an autoscaler intentionally reaches zero replicas,
the graph displays green even though the status model explicitly reserves neutral for intentional
scaled-to-zero workloads.
Code

pkg/topology/builder.go[R7904-7905]

+	default:
+		return StatusHealthy
Relevance

●●● Strong

Scaled-to-zero is explicitly neutral in the status model, so defaulting it healthy is a
deterministic mapping omission.

PR-#1024

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The diagnostic package emits StateScaledToZero when the dedicated condition is true and the
desired replica count is zero. The topology type documents StatusNeutral as the status for
intentional scaled-to-zero resources, but the new switch has no scaled-to-zero case and therefore
returns healthy.

pkg/hpadiag/diagnosis.go[118-121]
pkg/hpadiag/diagnosis.go[242-245]
pkg/topology/types.go[137-144]
pkg/topology/builder.go[7897-7905]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An autoscaler diagnosed as intentionally scaled to zero is classified as healthy rather than neutral.

## Issue Context
The topology status model explicitly defines neutral for intentional, idle, and scaled-to-zero resources. Add an explicit mapping for the corresponding autoscaler diagnosis and test it.

## Fix Focus Areas
- pkg/topology/builder.go[7897-7905]
- pkg/topology/node_health_test.go[110-163]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 41 rules
✅ Web pages:
  +2 more
Review mode: ⚖️ Balanced: This changes runtime topology health semantics across HPA and Gateway API paths, affecting operator and MCP health reporting, so it warrants a careful single-pass review.

Grey Divider

Tip of the day
💡 Did you know, you can copy the agent prompt from any finding and feed it to your IDE agent

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread pkg/topology/builder.go Outdated
Comment thread pkg/topology/builder.go
Comment thread pkg/topology/builder.go
@nadaverell
nadaverell force-pushed the fix/topology-status-false-claims branch from 20d8575 to dab49fa Compare September 7, 2026 15:00

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit dab49fa. Configure here.

Comment thread pkg/topology/builder.go
A node's status is not only a dot in the graph: MCP builds its
get_topology problem list from degraded and unhealthy nodes, and GitOps
derives managed-resource health from it. A green node is Radar answering
"nothing is wrong".

Gateway API routes were judged on Accepted alone. A gateway accepting a
route says nothing about whether the backends it forwards to exist, so a
route whose backendRef named a Service that was gone reported healthy —
the failure an operator opens the graph to find. Route health now reads
ResolvedRefs alongside Accepted, and ignores conditions a controller
observed against an earlier generation.

HorizontalPodAutoscalers were hardcoded healthy while the builder read
their replica counts three lines below. pkg/hpadiag already diagnoses
them for the drawer, so the node now uses it. An HPA that cannot reach
its metrics reports unhealthy instead of green. The mapping only ever
downgrades on negative evidence: routine scaling stays healthy, because
degraded nodes become reported problems and normal autoscaling is not
one.

Four Secret nodes are stubs synthesised from a reference — a TLS or CA
name mentioned by a Traefik or HTTPProxy resource that the Secret listing
never produced. They were built green, so a misspelled certificate
reference rendered as a healthy Secret. They report unknown: nothing
confirmed the Secret exists.
@nadaverell
nadaverell force-pushed the fix/topology-status-false-claims branch from dab49fa to edf6a40 Compare September 7, 2026 15:09
@nadaverell
nadaverell merged commit fb3e1ce into main Sep 7, 2026
9 checks passed
@nadaverell
nadaverell deleted the fix/topology-status-false-claims branch September 7, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant