Repository navigation
fix(e2e): caBundle verify race + apicompat update conflict flake - #443
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces CI flakiness in the e2e-test workflow by addressing a race where the controller’s cert-manager patches caBundle onto the ValidatingWebhookConfiguration asynchronously after the TLS secret becomes available.
Changes:
- Wrap the
caBundleinjection verification in a retry loop (up to 24 attempts with 5s sleep). - On final failure, emit the validating webhook configuration YAML and tail controller logs for easier debugging.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The "Verify controller RBAC and webhook-cert bootstrap" step checked the caBundle on rbgs-validating-webhook-configuration exactly once, immediately after the webhook-cert secret became present. The controller patches that caBundle asynchronously after creating the TLS secret, so the one-shot check races the controller and flakes (see failed run on PR sgl-project#441). Wrap the caBundle check in the same retry loop used for the webhook-cert secret (up to 24 x 5s), and dump controller logs on final failure. The bootstrap itself is fine; only the verification was too eager. Closes sgl-project#442
cheyang
force-pushed
the
fix/e2e-cabundle-retry
branch
from
August 29, 2026 14:27
76ad3e5 to
3d2f1ee
Compare
…flict The "rejects ... update that introduces a deprecated workload type" specs (RBG and RBGSet) did a single Get -> mutate -> Update. The controller writes status on freshly created objects, so its status patch can bump the resourceVersion between the test's Get and Update, surfacing a 409 conflict instead of reaching the validating webhook. containsRejection correctly treats a conflict as "not a rejection", so the spec fails even though the bootstrap is fine — a flake that passed on PR sgl-project#441's run but failed on sgl-project#443. Add updateWithConflictRetry, which re-Gets and retries on conflict (up to 10 attempts) and returns the final non-conflict error for the existing rejection assertion. Apply it to both update specs. See sgl-project#442
diw-zw
approved these changes
Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the e2e CI flakiness exposed by the v0.8.0 release PR (#441). The run failed in two independent places; both are pre-existing races, not v0.8.0 regressions.
Closes #442.
Flake 1 —
e2e-testjob: caBundle verify races the cert-manager patchThe "Verify controller RBAC and webhook-cert bootstrap" step in
.github/workflows/e2e-test.ymlfailed withcaBundle not injected into every webhook of rbgs-validating-webhook-configuration.But the controller patches
caBundleontorbgs-validating-webhook-configurationasynchronously, after creating the TLS secret. Timestamps from the failed run:patched caBundle on ValidatingWebhookConfigurationThe check ran ~95ms after the secret appeared; the controller finished patching ~110ms after the check. The controller logs confirm the patch succeeded moments later, so the bootstrap is fine — only the verification was too eager.
Fix: wrap the caBundle check in the same retry loop pattern used for the webhook-cert secret (up to 24 × 5s), and dump controller logs on final failure.
Flake 2 —
e2e-test-deprecated-disabledjob: apicompat update tests hit optimistic-concurrency conflictThe spec
rejects a RoleBasedGroupSet update that introduces a deprecated workload typefailed:The update specs (RBG and RBGSet) did a single
Get -> mutate -> Update. The controller writes status on freshly created objects, so its status patch bumps the resourceVersion between the test'sGetandUpdate, surfacing a 409 conflict instead of reaching the validating webhook.containsRejectioncorrectly treats a conflict as "not a rejection", so the spec flakes (it passed on PR #441's run, failed on this branch's run).Fix: add
updateWithConflictRetry, which re-Gets and retries on conflict (up to 10 attempts) and returns the final non-conflict error for the existing rejection assertion; apply it to both update specs intest/e2e/apicompat/specs_test.go.Both fixes must land together — neither flake is in the other's job, but each PR's CI runs the full e2e suite, so splitting them leaves each PR red on the other's flake.
Test plan
go test -c ./test/e2e/apicompat/compiles;go vetclean; gofmt clean.e2e-testjob: verify step passes even when the controller's caBundle patch lands after the TLS secret.e2e-test-deprecated-disabledjob: update specs pass even when the controller writes status between Get and Update.🤖 Generated with Claude Code