Skip to content

Commit e914705

Browse files
authored
Show Prometheus discovery progress, pre-warm on context switch, refuse headers without a URL (#1736)
## Why Users saw a **Discover Prometheus** button even though discovery is automatic. Two gaps: 1. Discovery already ran in the background at boot, but `prom.Status` had no in-flight state, so any metrics view opened during the 5–10 s run (4 s direct-probe budget + port-forward off-cluster) rendered the manual CTA. Clicking it only joined the run already underway. 2. After a context switch nothing started discovery at all — `Reinitialize` builds a fresh client and stops there. Separately: `--prometheus-header` values were sent to *every* auto-discovered candidate during probing. A mislabelled or hostile Service that looks like Prometheus would receive the operator's bearer token. ## What **Discovery status.** `prom.Status` gains `discovering`, owned by the detached discovery run itself — waiters come and go on their own deadlines, the flag follows the run (gate wait included). The last run's outcome, including a timeout, is reported as `error` when idle, so charts and Settings say *why* there is no connection instead of showing a bare button. Both are generation-keyed: an older run can never describe a newer configuration, and a cached-endpoint probe that finishes after a configuration change neither returns the superseded endpoint nor tears down what the new generation connected. **Pre-warm everywhere a cluster comes up.** `Prewarm()` runs at boot and from the context-switch and namespace-rescope callbacks. A cluster with no Prometheus no longer records an errorlog warning on every start; the status carries the reason. **Headers require a URL.** With headers configured and no URL, discovery refuses to run — in the Prometheus client and the Caretta traffic source — the status names the rule, and boot logs a warning (not fatal). `PUT /api/integrations/prometheus` rejects saving that pair, checked against what will actually be in effect: the submitted headers, else the running client's (flags, env and file already folded in), plus env-sourced headers on disk that re-resolve at the next start. The Helm chart's `values.schema.json` enforces the same dependency so `helm install`/`lint` fails on `traffic.prometheusHeaders*` without `traffic.prometheusUrl`. **Atomic live configuration.** A URL + header change applies in one step with the cached connection dropped alongside it (`prometheus.Configure`, `traffic.SetMetricsConfig`), and only the forward that served the dropped endpoint is stopped. The traffic subsystem is rebuilt on such a change — serialized with context operations — because a live source copies URL and headers at construction; without the rebuild, rotated or cleared credentials would keep being used until the next context switch. **Frontend.** Status polls every 2 s while a run is in flight or before the server's run has reported (a status read before the boot/switch pre-warm starts is transient — every run ends in `connected` or `error`); both chart CTA sites render "Discovering Prometheus…" instead of the button; `useAutoPromConnect` marks discovering optimistically before its POST and undoes it if the request itself fails, so an unreachable server can't leave the spinner up. Settings → Overview → Metrics shows Discovering / Connected / the failure reason. `@skyhook-io/k8s-ui` `PrometheusChartsView` gains an optional `discovering` prop (additive). ## Testing - **Go** (`internal/prometheus`, `internal/server`, `internal/traffic`, `pkg/prom`): status before / during / after a run; discovering follows the flight when every waiter cancels; failure outcome surfaces when idle; timeout and generation rules; headers gate never touches the cluster (fake client asserts no list); new credentials never reach the previous endpoint through `Configure`; a stale cached probe cannot tear down a newer connection; apply-handler 400 matrix (submitted / live / env-on-disk headers). **Vitest**: poll cadence. `helm lint` with and without headers/URL. - **Live, local Radar against an EKS cluster** with `monitoring/prometheus-operated`: opening a Pod drawer inside the boot window shows "Discovering Prometheus…" and flips to populated charts ~1 s later with no click; a header context switch to a cluster without Prometheus shows *Not connected — no Prometheus service found in cluster* in Settings, and switching back pre-warms and connects (~6 s) while the open drawer waits on the same run. Started with `--prometheus-header` and no URL: warning logged, no enumeration or probe, the drawer and Settings carry the rule, `PUT` with an empty URL returns 400, and clearing the headers in the same `PUT` discovers and connects. ## Notes - Local and in-cluster both pre-warm; in-cluster this is a `list services` plus in-cluster GETs, no port-forward. - A context switch performed server-side (API, CAPI connect) reaches an already-open tab on the regular 60 s status cadence; the header switcher invalidates queries itself. Unchanged by this PR. - Stacked follow-up: #1737 names the NetworkPolicy behind an in-cluster unreachable candidate. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes metrics discovery and how Prometheus auth headers are applied; misconfigurations that relied on headers without a URL will stop discovering until fixed, but the change reduces credential leakage to unintended endpoints. > > **Overview** > Improves **Prometheus metrics UX** and **credential safety** when Radar auto-discovers metrics backends. > > **Discovery progress** — `prom.Status` now exposes `discovering` (tied to the in-flight discovery run, not individual HTTP waiters) and surfaces the last failure (including timeouts) as `error` when idle. Boot and **context/namespace switches** call `Prewarm()` so discovery starts automatically; the UI polls every 2s while unsettled and shows “Discovering Prometheus…” instead of a manual CTA. > > **Headers require an explicit URL** — Shared rule via `pkg/prom.HeadersRequireURL` / `ErrHeadersRequireURL`: with auth headers but no `prometheusUrl`, discovery is skipped (Prometheus client and Caretta traffic), Settings/`PUT /api/integrations/prometheus` reject saving that combo, and Helm `values.schema.json` enforces `traffic.prometheusUrl` when headers are set. Docs call out that credentials are never sent to auto-discovered endpoints. > > **Safer live config** — URL and headers apply atomically (`Configure`, `SetMetricsConfig`) with the cached connection dropped in the same step; live Prometheus integration changes **restart the traffic subsystem** so rotated headers cannot keep hitting an old endpoint. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 52216a7. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 11278fa commit e914705

22 files changed

Lines changed: 784 additions & 81 deletions

‎deploy/helm/radar/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -207,8 +207,8 @@ trend charts remain unavailable for Kubecost.
207207
| `cost.kubecost.existingSecret` | Secret holding an optional Kubecost service-account API key; setting it disables automatic port-9008 auth bypass | `""` |
208208
| `cost.kubecost.existingSecretKey` | Key within `cost.kubecost.existingSecret`; sent as `X-API-KEY` | `api-key` |
209209
| `traffic.prometheusUrl` | Manual Prometheus/VictoriaMetrics URL (skips auto-discovery) | `""` |
210-
| `traffic.prometheusHeaders` | HTTP headers sent with every Prometheus request (auth-protected backends) | `{}` |
211-
| `traffic.prometheusHeadersFromEnv` | Prometheus headers sourced from environment variables, for secret-backed auth headers | `{}` |
210+
| `traffic.prometheusHeaders` | HTTP headers sent with every Prometheus request (auth-protected backends). Requires `traffic.prometheusUrl` — credentials are never sent to auto-discovered endpoints | `{}` |
211+
| `traffic.prometheusHeadersFromEnv` | Prometheus headers sourced from environment variables, for secret-backed auth headers. Requires `traffic.prometheusUrl` | `{}` |
212212
| `argocd.existingSecret` | Name of a Secret holding the Argo CD API token (recommended — keeps it out of the release) | `""` |
213213
| `argocd.existingSecretKey` | Key within `argocd.existingSecret` holding the token | `token` |
214214
| `argocd.token` | Inline Argo CD API token (dev only — lands in the release state) | `""` |

‎deploy/helm/radar/values.schema.json‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -362,7 +362,17 @@
362362
"type": "object",
363363
"additionalProperties": { "type": "string" }
364364
}
365-
}
365+
},
366+
"allOf": [
367+
{
368+
"if": { "properties": { "prometheusHeaders": { "minProperties": 1 } }, "required": ["prometheusHeaders"] },
369+
"then": { "properties": { "prometheusUrl": { "pattern": "\\S" } }, "required": ["prometheusUrl"] }
370+
},
371+
{
372+
"if": { "properties": { "prometheusHeadersFromEnv": { "minProperties": 1 } }, "required": ["prometheusHeadersFromEnv"] },
373+
"then": { "properties": { "prometheusUrl": { "pattern": "\\S" } }, "required": ["prometheusUrl"] }
374+
}
375+
]
366376
},
367377
"argocd": {
368378
"type": "object",

‎deploy/helm/radar/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,8 @@ traffic:
533533
prometheusUrl: ""
534534
# HTTP headers sent with every Prometheus request. Required for
535535
# auth-protected backends (Bearer tokens, X-Scope-OrgID multi-tenancy, etc.).
536+
# Setting headers requires prometheusUrl: Radar never sends credentials to
537+
# auto-discovered endpoints, and the chart schema rejects the combination.
536538
# Values are rendered into the pod spec; use prometheusHeadersFromEnv for
537539
# secrets.
538540
# Example:

‎docs/configuration.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,11 +111,11 @@ All fields are optional — omitted fields use built-in defaults.
111111
| `kubecostUrl` | Optional Kubecost 3 Aggregator base URL. Empty discovers an active local Aggregator Service and tries its named `tcp-api` port (9004). When that port requires SAML/OIDC and no API key is configured, Radar can fall back to the same Service's exact `tcp-api-rbac` port (9008). Federated agent-only clusters need the central URL; root API URLs and URLs ending in `/model` are accepted. |
112112
| `kubecostClusterId` | Cluster ID used to filter a central Aggregator. Empty detects one distinct literal `CLUSTER_ID` from an active FinOps Agent or Aggregator; indirect or conflicting values require an override. An override saved in Settings is bound to the active kubeconfig context so switching clusters cannot silently reuse the wrong cluster's costs. A value added directly to the config file is bound and persisted on its first startup with an available kubeconfig context. |
113113
| `kubecostApiKey` | Optional Kubecost service-account key sent as `X-API-KEY`. Stored in the unencrypted `0600` config file and redacted from `GET /api/config`; changing the URL origin clears a stored key unless it is supplied again. With a blank URL, a key saved in Settings is bound to the active kubeconfig context because Radar will auto-discover that cluster's local Aggregator; a key added directly to the config file is bound and persisted on its first startup with an available kubeconfig context. An explicit key is never bypassed through an auto-discovered unauthenticated port: authentication failure remains visible. A key paired with an explicit central Aggregator URL can be reused across contexts. In the Helm deployment, the UI-written config lives on the pod's temporary `emptyDir` and does not survive pod replacement; use a Kubernetes Secret with `cost.kubecost.existingSecret` instead. |
114-
| `prometheusHeaders` | HTTP headers sent with every Prometheus request. Required for auth-protected backends — e.g. `{"X-Scope-OrgID": "my-org"}`. Equivalent CLI: `--prometheus-header Key=Value` (repeatable). Stored in plain text in `config.json` — protect the file accordingly. |
114+
| `prometheusHeaders` | HTTP headers sent with every Prometheus request. Required for auth-protected backends — e.g. `{"X-Scope-OrgID": "my-org"}`. Equivalent CLI: `--prometheus-header Key=Value` (repeatable). Stored in plain text in `config.json` — protect the file accordingly. **Requires `prometheusUrl`**: headers carry credentials, and auto-discovery probes every Service that looks like Prometheus, so with headers configured and no URL Radar refuses to discover (the Metrics status names the rule) rather than send them to endpoints you never named. Settings rejects saving that combination. |
115115
| `argoCdUrl` | Manual argocd-server URL for the Argo CD API integration — skips auto-discovery. |
116116
| `argoCdToken` | Argo CD API token (get-only account recommended). Stored in plain text — the file is written `0600`; the token is redacted from `GET /api/config`. |
117117
| `argoCdInsecureTls` | Skip TLS verification for argocd-server (self-signed default installs). Scoped to the Argo CD client only. |
118-
| `prometheusHeadersFromEnv` | Header values read from environment variables at startup — e.g. `{"Authorization": "PROMETHEUS_TOKEN"}`. Equivalent CLI: `--prometheus-header-from-env Key=ENV_VAR` (repeatable). Use this with Kubernetes Secret-backed env vars in Helm deployments. |
118+
| `prometheusHeadersFromEnv` | Header values read from environment variables at startup — e.g. `{"Authorization": "PROMETHEUS_TOKEN"}`. Equivalent CLI: `--prometheus-header-from-env Key=ENV_VAR` (repeatable). Use this with Kubernetes Secret-backed env vars in Helm deployments. Same rule as `prometheusHeaders`: requires `prometheusUrl`. |
119119
| `mcp` | Enable/disable MCP server for AI tools (default: enabled) |
120120
| `debugImage` | Image for ephemeral debug containers and node debug pods (same as `--debug-image`). Empty = `busybox:latest`; point at a mirror for air-gapped / private-registry clusters. |
121121

‎docs/in-cluster.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -522,8 +522,8 @@ See [Helm Chart README](../deploy/helm/radar/README.md) for all available values
522522
| `cost.kubecost.url` | Kubecost 3 Aggregator URL; blank discovers local port 9004 and may fall back to the named SAML/OIDC bypass port 9008 without a key; required for agent-only clusters | `""` (discover local) |
523523
| `cost.kubecost.clusterId` | Cluster ID filter for a central Aggregator | `""` (detect literal `CLUSTER_ID`) |
524524
| `cost.kubecost.existingSecret` | Secret containing an optional Kubecost API key; setting it disables automatic port-9008 auth bypass | `""` |
525-
| `traffic.prometheusUrl` | Manual PromQL-compatible query URL (Prometheus, VictoriaMetrics, Thanos, Mimir) | `""` (auto-discover) |
526-
| `traffic.prometheusHeadersFromEnv` | Prometheus headers sourced from environment variables, for secret-backed auth headers | `{}` |
525+
| `traffic.prometheusUrl` | Manual PromQL-compatible query URL (Prometheus, VictoriaMetrics, Thanos, Mimir). Required whenever headers are set | `""` (auto-discover) |
526+
| `traffic.prometheusHeadersFromEnv` | Prometheus headers sourced from environment variables, for secret-backed auth headers. Requires `traffic.prometheusUrl` — credentials are never sent to auto-discovered endpoints | `{}` |
527527
| `persistence.enabled` | Enable PVC for SQLite storage | `false` |
528528
| `persistence.size` | PVC size | `1Gi` |
529529
| `rbac.podLogs` | Enable log viewer | `true` |
@@ -557,6 +557,14 @@ kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
557557

558558
If the UI loads through an ingress but pod exec does not connect, ensure the ingress forwards WebSocket upgrades and preserves the browser-facing `Host` header. Preserving `Host` is the compatibility requirement across browsers and proxies; Fetch Metadata is an additional signal only when both sides forward it. Radar logs rejected handshakes with both `Origin` and `Host`.
559559

560+
### Metrics charts show "Prometheus not connected"
561+
562+
Radar discovers Prometheus in the background as soon as it connects to the cluster (and again after every context switch), by listing Services and probing the ones that look like a PromQL backend at their in-cluster address. While that runs the charts show "Discovering Prometheus…"; once it ends without a connection, the charts and **Settings → Overview → Metrics** show why:
563+
564+
- *no Prometheus service found in cluster* — nothing matched the well-known names or labels. Set `traffic.prometheusUrl`.
565+
- *no Prometheus service reachable in cluster* — a candidate exists but its Service address did not answer from Radar's pod. In-cluster Radar does not fall back to port-forwarding, so a NetworkPolicy or mesh rule that blocks Radar's namespace from the monitoring namespace shows up exactly like this. Allow ingress from Radar's namespace, or point `traffic.prometheusUrl` at an address that is reachable.
566+
- *Prometheus headers are configured but no Prometheus URL is set* — see `traffic.prometheusHeadersFromEnv` above.
567+
560568
### Basic auth prompt not appearing
561569

562570
Verify the secret format:

‎internal/app/bootstrap.go‎

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ import (
2929
"github.com/skyhook-io/radar/internal/timeline"
3030
"github.com/skyhook-io/radar/internal/traffic"
3131
versionpkg "github.com/skyhook-io/radar/internal/version"
32+
"github.com/skyhook-io/radar/pkg/prom"
3233
)
3334

3435
var clusterConnectionProbe = k8s.TestClusterConnection
@@ -292,6 +293,9 @@ func RegisterCallbacks(cfg AppConfig, timelineStoreCfg timeline.StoreConfig) {
292293
if len(cfg.PrometheusHeaders) > 0 {
293294
traffic.SetMetricsHeaders(cfg.PrometheusHeaders)
294295
prometheuspkg.SetHeaders(cfg.PrometheusHeaders)
296+
if prom.HeadersRequireURL(cfg.PrometheusURL, cfg.PrometheusHeaders) {
297+
log.Printf("[prometheus] Warning: %v", prom.ErrHeadersRequireURL)
298+
}
295299
}
296300
cfg = persistKubecostContextBindings(cfg)
297301
if err := internalopencost.ConfigureStartup(internalopencost.ManagerConfig{
@@ -319,6 +323,11 @@ func RegisterCallbacks(cfg AppConfig, timelineStoreCfg timeline.StoreConfig) {
319323
prometheuspkg.Reinitialize(k8s.GetClient(), k8s.GetConfig(), k8s.GetContextName())
320324
return nil
321325
})
326+
// Reinitialize only builds the new client; discovery for the new cluster has
327+
// to be started explicitly, and the callback fires once subsystem init is
328+
// done so the run sees a populated cache.
329+
k8s.OnContextSwitch(func(_ string) { prometheuspkg.Prewarm() })
330+
k8s.OnNamespaceRescope(func(_ string) { prometheuspkg.Prewarm() })
322331
k8s.RegisterCostResetFunc(internalopencost.Reset)
323332
}
324333

@@ -578,21 +587,7 @@ func InitializeCluster() {
578587
ClusterName: k8s.GetClusterName(),
579588
})
580589

581-
// Auto-discover Prometheus in the background so charts are ready immediately
582-
go func() {
583-
pt := time.Now()
584-
promCtx, promCancel := context.WithTimeout(context.Background(), 30*time.Second)
585-
defer promCancel()
586-
client := prometheuspkg.GetClient()
587-
if client == nil {
588-
return
589-
}
590-
if _, _, err := client.EnsureConnected(promCtx); err != nil {
591-
log.Printf("[prometheus] Auto-discovery failed (%v): %v", time.Since(pt), err)
592-
} else {
593-
log.Printf("[prometheus] Auto-discovery succeeded (%v)", time.Since(pt))
594-
}
595-
}()
590+
prometheuspkg.Prewarm()
596591
}
597592

598593
// mcpPortFileDisabled suppresses port-file writes AND removals — an ephemeral

‎internal/k8s/context_manager.go‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,32 @@ func RegisterTrafficFuncs(reset TrafficResetFunc, reinit TrafficReinitFunc) {
268268
trafficReinitFunc = reinit
269269
}
270270

271+
// RestartTrafficSubsystem tears down and rebuilds the traffic manager for the
272+
// current cluster, the way a context switch does, serialized with any switch
273+
// or rescope in flight so the two can't interleave on the manager singleton.
274+
// A live configuration change that the traffic sources copied at construction
275+
// (the metrics URL and headers) reaches them only this way. No-op while
276+
// disconnected: the next connect builds the manager from current config.
277+
func RestartTrafficSubsystem() error {
278+
contextSwitchMu.RLock()
279+
resetFn, reinitFn := trafficResetFunc, trafficReinitFunc
280+
contextSwitchMu.RUnlock()
281+
if resetFn == nil || reinitFn == nil {
282+
return nil
283+
}
284+
activeContextOperations.Add(1)
285+
contextOpMu.Lock()
286+
defer func() {
287+
activeContextOperations.Add(-1)
288+
contextOpMu.Unlock()
289+
}()
290+
if GetClient() == nil {
291+
return nil
292+
}
293+
resetFn()
294+
return reinitFn()
295+
}
296+
271297
// RegisterPrometheusFuncs registers the Prometheus client reset/reinit functions.
272298
func RegisterPrometheusFuncs(reset PrometheusResetFunc, reinit PrometheusReinitFunc) {
273299
contextSwitchMu.Lock()

‎internal/prometheus/availability.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ import (
44
"context"
55
"errors"
66
"strings"
7+
8+
"github.com/skyhook-io/radar/pkg/prom"
79
)
810

911
// AvailabilityStatus is the coarse answer to "can Radar reach Prometheus
@@ -83,7 +85,9 @@ func (c *Client) Availability(ctx context.Context) AvailabilityState {
8385
// caller's context ends mid-probe: the endpoint is known and only
8486
// this probe failed.
8587
return AvailabilityState{State: AvailabilityConfiguredFailed, Address: baseURL, Err: err}
86-
case errors.Is(err, errPrometheusUnreachable):
88+
case errors.Is(err, errPrometheusUnreachable), errors.Is(err, prom.ErrHeadersRequireURL):
89+
// Headers without a URL is an operator configuration that discovery
90+
// refuses to act on — configured, and failing for a stated reason.
8791
return AvailabilityState{State: AvailabilityConfiguredFailed, Err: err}
8892
case !hasK8s,
8993
errors.Is(err, ErrPrometheusNotFound),

0 commit comments

Comments
 (0)