Skip to content

Helm actions: change objects in the release's namespace, not where its history is stored - #2080

Merged
nadaverell merged 3 commits into
mainfrom
fix/helm-actions-target-namespace
Oct 13, 2026
Merged

nadaverell merged 3 commits into
mainfrom
fix/helm-actions-target-namespace

Conversation

@nadaverell

@nadaverell nadaverell commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Helm actions on a Flux-managed release changed copies of its objects in the wrong namespace instead of the release itself, and every action reported success. This makes rollback, upgrades, values edits and uninstall in Radar's Helm view act on the release's real objects.

The problem. Flux's helm-controller stores a release's Helm history in one namespace (by default flux-system, next to the HelmRelease) and deploys its objects into spec.targetNamespace, say payments. Radar built each Helm action's Kubernetes client for the storage namespace. A chart object without metadata.namespace, which includes everything from the helm create scaffold, therefore went to flux-system:

  • Values apply and upgrades created a second copy of the workload in flux-system; the real one in payments stayed unchanged.
  • Rollback rolled back that copy, not the real workload.
  • Flux's next reconcile kept the real objects and never cleaned up the copy.
  • Uninstall deleted the copy and the release history, and left the real workload running, tracked by no release.

Radar already marks these releases "Managed by Flux" and warns that direct changes are reverted at the next reconcile. That warning only holds once the action reaches the real objects.

What changed

Helm actions use the release's recorded namespace for objects. Before each action, useReleaseTargetNamespace points Helm's object builder at the namespace recorded on the release. Storage stays where it is, and objects that name a namespace keep it. The action configuration is built fresh for every call, so nothing carries over to the next request, and impersonation is unchanged; the API server checks the user's rights in the namespace actually written. This covers:

  • rollback, with and without progress, including the hooks it runs;
  • chart-version upgrade, with and without progress;
  • upgrade with edited values;
  • values preview and values apply;
  • uninstall;
  • install recovery (re-installing over a failed release).

In Helm 3.22, upgrades already render .Release.Namespace from the existing release and record that namespace on the new revision (pkg/action/upgrade.go 256 and 288), so the rendered and built namespaces now agree. Fresh installs, reinstalls and reads are unchanged. docs/helm.md notes the split.

Cloud Connect refuses to adopt a Radar release split this way. Adoption checks permissions, creates the token Secret and reads the Deployment in the namespace that holds the history. For a release running elsewhere that never worked: it started a second Radar next to the history and left the original unconnected. Preparation now stops and says to upgrade the release with the tool that installed it. The identity Cloud Connect rechecks before writing, and when verifying a rollback, includes the recorded namespace, so a record moved after approval doesn't pass as unchanged. A record with no recorded namespace is not treated as split.

This fix was split out of #2046 so it can ship without that PR's MCP Helm tools. #2046 will be restacked onto it and drop its copy of the helper.

Notes for review

  • The change is the single helper in internal/helm/client.go and where it is called. It mutates only the per-call kube.Client's default namespace, not the storage driver or the REST getter.
  • A release record with no namespace keeps today's behaviour: Helm's client falls back to the storage namespace, which is right for an ordinary single-namespace release. Standard Helm always records the namespace.
  • Upgrades take the namespace from the latest record. Helm's own upgrade and rollback carry the namespace forward, so only hand-edited history could make revisions disagree.

Testing

  • Live on EKS (Kubernetes 1.34) with Flux 2.9. Two Flux HelmReleases of Helm's hello-world example chart, with history in flux-system and objects in their own namespace, each at revision 2 and suspended. Radar built from main and from this branch each took one release through values preview, values apply (3 replicas), rollback to revision 1, a Flux resume-and-reconcile, and uninstall. Every Radar call returned 200 on both builds.

    Step main this branch
    Values apply New Deployment, Service and ServiceAccount in flux-system at 3 replicas; the real Deployment stays at 2 The real Deployment goes to 3; nothing in flux-system
    Rollback The flux-system copy goes back to 1; the real Deployment stays at 2 The real Deployment goes back to 1
    Flux reconcile The real Deployment stays at 2; the copy is left behind Flux restores 2 on the real objects
    Uninstall Copy and history deleted; the real workload keeps running Real objects and history deleted; nothing left

    On both builds the history Secrets stayed in flux-system. The test namespaces and HelmReleases were deleted afterwards.

  • Live on envtest (Kubernetes 1.36.2): values preview and apply, rollback and uninstall show the same main-vs-branch difference.

  • internal/helm/release_target_namespace_test.go: nine cases run the real Helm SDK actions against a recording API server, with history in store. Each checks that:

    • objects without a namespace go to app, and explicit namespaces stay put;
    • history stays in store;
    • a resource-policy: keep object survives uninstall;
    • pre-rollback hooks go to the right namespace;
    • upgrade cases render .Release.Namespace as app.

    All nine fail with the production change reverted.

  • TestPrepareCloudUpgradeRefusesReleaseRunningOutsideItsStorageNamespace covers the refusal, and that a record with no namespace isn't refused. TestPreparedReleaseIdentityRollbackMatchIgnoresOnlyRevision covers a record that moved namespace.

  • Checks: go build ./..., go test ./internal/helm/... ./internal/cloudinstall/..., the Helm route tests in internal/server; golangci-lint --new-from-rev reports 0 issues.

  • Not run live: chart-version upgrades, which the unit cases cover against a local chart repository, and Cloud Connect adoption.

  • No UI change, so no visual test.

â€Ķs history is stored

Flux and `helm --storage-namespace` keep a release's history in one
namespace (e.g. flux-system) and its objects in another. Radar built every
action's Kubernetes client for the storage namespace, so chart objects
without metadata.namespace were written there instead: applying values or
rolling back created duplicate workloads in flux-system and reported
success, and uninstall left the real workloads running.

Rollback, upgrade, edited-values upgrade, values preview and apply,
uninstall, install recovery and the prepared upgrade now build objects in
the release's recorded namespace; history stays where it is stored and
explicit namespaces are kept.
â€Ķhistory's namespace

Adoption checks permissions, creates the token Secret and reads the
Deployment in the namespace that holds the release history. For a release
running elsewhere that never worked: it started a second Radar next to the
history and left the original unconnected. Preparation now stops with what
to do instead, and the prepared dry-run no longer needs the namespace
routing. Rollback tests now also cover pre-rollback hooks, which went to
the storage namespace too.
The rechecks before writing and the rollback verification compare a pinned
release identity. It now includes the recorded namespace, so a record moved
to another namespace after approval no longer passes as unchanged. A record
with no namespace is not treated as split, as before.
@nadaverell
nadaverell merged commit 22379df into main Oct 13, 2026
9 checks passed
@nadaverell
nadaverell deleted the fix/helm-actions-target-namespace branch October 13, 2026 18:14
nadaverell added a commit that referenced this pull request Oct 18, 2026
# Fix data race in the helm target-namespace test server

`targetNamespaceActionConfig` (added in #2080) starts an `httptest`
server whose handler reads and writes a shared `objects` map and appends
to a `writes` slice. The kube client sends requests concurrently, so `go
test` sometimes died with `fatal error: concurrent map read and map
write`, as in the Backend job on #2198.

The handler now holds a `sync.Mutex` while it touches that state. Only
the test changes.

**Verification** (`go test -race -count=20 ./internal/helm -run
TargetNamespace`):
- Before: FAIL, with 3 `WARNING: DATA RACE` reports.
- After: `ok` (20.2s), no race reports.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant