Skip to content

Tighten unused ConfigMap and Secret audit checks - #1067

Merged
nadaverell merged 8 commits into
mainfrom
nadav/sky-1100-unused-config-precision
Jul 5, 2026
Merged

nadaverell merged 8 commits into
mainfrom
nadav/sky-1100-unused-config-precision

Conversation

@nadaverell

@nadaverell nadaverell commented Jun 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

The Unused ConfigMap or Secret audit check can lead an operator toward deletion, so false positives are the dangerous failure mode. It previously credited direct Pod refs and a few controller refs, but missed several real consumption patterns: workload templates, ServiceAccount pull-secret defaults, supported CRD refs, platform/controller-owned artifacts, dynamic sidecar-loaded ConfigMaps, leader-election locks, managed add-on Secrets, and selected controller config objects consumed by convention.

Linear: SKY-1100

What changed

  • Treats active Deployment, StatefulSet, DaemonSet, Job, and CronJob pod templates as ConfigMap/Secret consumers.
  • Stops treating terminal Jobs and Succeeded/Failed Pods as active consumers, so completed one-off workload artifacts do not hide stale ConfigMaps/Secrets indefinitely.
  • Credits ServiceAccount imagePullSecrets when a PodSpec relies on ServiceAccount admission defaults, including the implicit default ServiceAccount.
  • Includes ephemeral-container env/envFrom refs, CSI nodePublishSecretRef, and legacy in-tree volume secret refs in the consumed-resource index.
  • Adds a cache-only CRD reference extractor for supported integrations. It reads WatchedGVRs/ListWatched only and does not start new informers from the audit request path.
  • Counts explicit supported-controller ConfigMap/Secret refs for Gateway API, Traefik, Contour, Istio Gateway TLS credentials, cert-manager Issuer/ClusterIssuer ACME refs, Flux, External Secrets, KEDA TriggerAuthentication, Prometheus Operator monitors/Alertmanager, Crossplane provider-kubernetes/provider-helm, Argo Rollouts, Knative templates, CNPG, CAPI KubeadmConfig, and Velero storage locations.
  • Handles context-sensitive supported refs where the CRD object alone is not enough: Crossplane HelmRelease selectors default to the release namespace, Rollout/Knative pod templates inherit ServiceAccount imagePullSecrets when they omit explicit pull secrets, and cert-manager ClusterIssuer secrets use the cached cert-manager controller's cluster-resource namespace when it can be inferred.
  • Adds cert-manager Issuer/ClusterIssuer, CNPG, CAPI bootstrap, and Velero storage-location kinds to the supported CRD warmup list, so those refs do not depend on generic discovery's small-CRD eager-watch heuristic.
  • Skips controller-owned ConfigMaps/Secrets for the orphan check because another resource owns their lifecycle.
  • Skips known platform/controller artifacts that are consumed by convention rather than Kubernetes object refs, including Kubernetes/EKS/GKE control-plane ConfigMaps, Argo CD fixed config, ingress-nginx config, sealed-secrets key material, cert-manager webhook CA artifacts, managed-monitoring/controller artifacts found in live review, sidecar-loaded Grafana/Prometheus/fluentd/K8sGPT ConfigMaps, leader-election lock ConfigMaps, Argo Workflows and CNPG controller config ConfigMaps with controller labels, and a narrow GKE managed-prometheus gmp-system/webhook-tls Secret shape.
  • Avoids generic field-name scanning and skips ambiguous/unbounded cases such as arbitrary Crossplane managed-resource manifests, Crossplane connection-output fields, cluster-scoped auth refs with unclear namespace semantics, name-only dashboard guesses, and controller command-line conventions without a precise supported-integration rule.
  • Tightens secretInConfigMap so broad keys like auth, token, secret, and credentials require a secret-looking value. Low-information settings such as auth_mode=oidc, token_ttl=3600, credentials file paths, Secret-name references, and non-credential URLs no longer fire; credential URLs with username or password userinfo still do.

Testing

  • go test ./internal/audit -run 'TestDynamicConfigObjectRefs|TestListDynamicConfigObjectRefsFiltersByTargetNamespace'
  • go test ./internal/k8s -run 'TestChartGrantsEveryWatchedCRDGroup|TestCapabilitiesAlignment'
  • cd pkg && go test ./audit -run 'TestOrphanConfigMapSecret|TestSecretInConfigMap'
  • cd pkg && go test ./audit
  • make test
  • make build
  • Final focused validation: cd pkg && go test ./audit -run 'TestOrphanConfigMapSecret'
  • Expanded regression coverage with the full live-derived platform/controller artifact corpus plus negative cases that keep same-looking ordinary objects flagged.
  • Added contextual CRD regression tests for Crossplane same-namespace selectors, ServiceAccount pull-secret defaults in CRD pod templates, and custom cert-manager cluster-resource namespaces.
  • Added dynamic-loader, controller-config, and managed-add-on regression tests for Grafana dashboard/datasource labels, Prometheus rule labels, fluentd loader labels, K8sGPT dynamic labels, leader-election annotations, Argo Workflows controller config labels, CNPG controller config labels, and GKE managed-prometheus webhook TLS metadata, with near-miss negative cases for false labels, empty annotations, unlabeled dashboard-looking ConfigMaps, same-name controller ConfigMaps without controller labels, and same-name Secrets outside the managed namespace.
  • visual-test: skipped; this is backend audit precision with no UI/layout/browser behavior change.

Live validation

  • Ran the built PR binary against internal GKE and EKS clusters. Radar started, dynamic warmup synced, /api/audit?raw=true returned valid audit JSON, and the orphan check stayed nonzero rather than being blanket-suppressed.
  • Verified live Gateway certificateRefs on three GKE clusters: 7 total live Gateway Secret refs checked, 0 were incorrectly reported as orphaned.
  • Reviewed live orphan findings by namespace/name pattern and inspected representative object metadata. Removed rows mapped to verified platform/controller artifacts, controller labels/annotations, or structured CRD references.
  • Post-fix radar-test EKS reruns with explicit namespace lists:
    • nonprod: 194 -> 189 total findings; orphanConfigMapSecret 12 -> 7; known platform rows checked: 0 remaining.
    • management: 154 -> 146 total findings; orphanConfigMapSecret 10 -> 2; known platform rows checked: 0 remaining.
    • prod: 7488 -> 7484 total findings; orphanConfigMapSecret 885 -> 881; known platform rows checked: 0 remaining.
  • Post-fix GKE management rerun: 875 -> 808 total findings; orphanConfigMapSecret 89 -> 22; known platform/cert-manager issuer rows checked: 0 remaining.
  • Final GKE nonprod validation:
    • Confirmed live Grafana dashboard/datasource ConfigMaps carry the expected sidecar labels.
    • Confirmed live leader-election ConfigMaps carry control-plane.alpha.kubernetes.io/leader.
    • Confirmed live gmp-system/webhook-tls carries GKE managed-prometheus metadata.
    • Confirmed live Argo Workflows and CNPG controller ConfigMaps carry the expected controller labels.
    • The targeted dashboard/datasource, leader-lock, GKE managed-prometheus, Argo Workflows controller config, and CNPG controller config resources did not appear in live orphanConfigMapSecret results.
    • An unlabeled dashboard-looking ConfigMap still appeared as an orphan finding, which preserves the intended no-name-only-guess guardrail.
  • Reachable cloud clusters had ExternalSecrets/CNPG CRDs installed in some cases but no live ExternalSecret, SecretStore, or CNPG Cluster/Pooler objects to validate those structured reference paths; those remain covered by table/unit tests and source-shape review.

Risk / blast radius

Blast radius is limited to audit finding precision for ConfigMap/Secret cleanup and ConfigMap-secret warnings. No resources are mutated, and no topology/cache/API write behavior changes.

The main risk is over-suppression: hiding a real unused ConfigMap/Secret because it happens to match a platform, dynamic-loader, or controller-config shape. Mitigations are exact metadata keys, namespace/name gates where needed, managed-add-on evidence for the GKE Secret, controller labels for Argo Workflows/CNPG config, and negative tests for same-looking ordinary objects.

Runtime cost remains bounded. The audit scan itself stays cache-only: it checks an explicit handler table before reading cached objects, and it never calls EnsureWatching, discovery, or live Kubernetes List() from the request path. The added apiserver/watch cost is startup warmup for a small set of Radar-supported integration CRDs when those CRDs are installed.

Product tradeoff: secretInConfigMap is now more precision-biased, so very short real secrets under ambiguous keys may be missed. Terminal Jobs and completed Pods no longer suppress orphan findings; this assumes completed workload artifacts are historical rather than active consumers. Some ambiguous controller-output, non-modeled chart-internal config, and Istio DestinationRule TLS credential namespace cases remain intentionally uncovered rather than guessed.

@nadaverell
nadaverell requested a review from hisco as a code owner June 30, 2026 02:10
Comment thread internal/audit/runner.go
@nadaverell
nadaverell force-pushed the nadav/sky-1100-unused-config-precision branch 4 times, most recently from 275427e to bf09d52 Compare June 30, 2026 21:44
@nadaverell
nadaverell force-pushed the nadav/sky-1100-unused-config-precision branch from bf09d52 to b71e516 Compare July 4, 2026 14:01
Comment thread internal/audit/crd_config_refs.go
Comment thread internal/audit/crd_config_refs.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0af1464. Configure here.

Comment thread internal/audit/crd_config_refs.go
@nadaverell
nadaverell merged commit 8bff5ee into main Jul 5, 2026
9 checks passed
@nadaverell
nadaverell deleted the nadav/sky-1100-unused-config-precision branch July 5, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant