Skip to content

Fix intermittent "context canceled" token failures in managed identity retry() - #642

Merged
Nilesh Choudhary (4gust) merged 2 commits into
mainfrom
4gust-silver-succotash
Jul 30, 2026
Merged

Nilesh Choudhary (4gust) merged 2 commits into
mainfrom
4gust-silver-succotash

Conversation

@4gust

@4gust Nilesh Choudhary (4gust) commented Jul 28, 2026 •

Copy link
Copy Markdown
Collaborator

What's going on

Fixes #634.

Managed identity token acquisitions were intermittently failing with context canceled right after a successful HTTP 200, especially when the identity endpoint (IMDS/App Service) was responding slowly. That is exactly when the retry loop kicks in, so the bug showed up precisely in the situation the retry logic is supposed to handle.

The root cause

Inside retry() in apps/managedidentity/managedidentity.go, each attempt created a per-attempt context with defer tryCancel() inside the loop. Because defer runs when the function returns (not at the end of each iteration), every stacked cancel fired the moment retry() returned, including the one tied to the response body that was being handed back to the caller.

The caller then did io.ReadAll(resp.Body) on a body whose context had already been canceled. Whether that read succeeded was a race against the transport having buffered the body. Small token JSON over a fast connection usually won the race, which is why this was intermittent and got much worse when the endpoint was slow.

The fix

  • Buffer the response body into memory (small in-memory reader) while the attempt's context is still alive, on both the success path and the final-attempt path, then cancel. Once the body is buffered, canceling the context can no longer break the caller's read.
  • Manage the per-attempt cancels explicitly instead of stacking defer calls in the loop. The previous attempt's context is canceled only after its body has been drained, so connections can still be reused. This also clears up the classic "defer inside a loop" resource retention that was called out in the issue.

Tests

  • Added TestRetrySlowBodyDoesNotCancelContext, a regression test using a real httptest server that returns a retryable 500 first and then a slow body. It fails on the old code with context canceled and passes with the fix.
  • Added TestRetryPolicyDisabledSlowBody to confirm the single-request (retry disabled) path handles the same slow body, isolating the bug to retry().
  • Verified the existing TestRetryFunction and Test_RetryPolicy_For_AcquireToken still pass, including the "max retries reached" case that reads the body of a retryable response.
  • Cross-referenced against the exact repro from the issue (run as an external module against this branch): it reproduces context canceled on the pre-fix code and passes after the fix.

retry() deferred every per-attempt context cancel until it returned, so
the context bound to the returned resp.Body was already canceled when the
caller read it. On slow identity-endpoint responses the caller's io.ReadAll
raced the cancel and surfaced 'context canceled' on a successful HTTP 200.

Buffer the response body into memory while the attempt context is still
alive on both the success and final-attempt paths, and manage per-attempt
cancels explicitly (canceling the previous attempt after its body is
drained) instead of stacking defers in the loop.

Fixes #634

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c006218-507e-4cf9-8cdb-d12204c25b5c
@sonarqubecloud

Copy link
Copy Markdown

@4gust
Nilesh Choudhary (4gust) merged commit b434989 into main Jul 30, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] managedidentity: retry() cancels the response body's context before the caller reads it, causing intermittent "context canceled" token failures

4 participants