Skip to content

fix(e2e): caBundle verify race + apicompat update conflict flake - #443

Merged
diw-zw merged 2 commits into
sgl-project:mainfrom
cheyang:fix/e2e-cabundle-retry
Aug 29, 2026
Merged

diw-zw merged 2 commits into
sgl-project:mainfrom
cheyang:fix/e2e-cabundle-retry

Conversation

@cheyang

@cheyang cheyang commented Aug 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the e2e CI flakiness exposed by the v0.8.0 release PR (#441). The run failed in two independent places; both are pre-existing races, not v0.8.0 regressions.

Closes #442.

Flake 1 — e2e-test job: caBundle verify races the cert-manager patch

The "Verify controller RBAC and webhook-cert bootstrap" step in .github/workflows/e2e-test.yml failed with caBundle not injected into every webhook of rbgs-validating-webhook-configuration.

  • The webhook-cert secret check retries in a loop (up to ~120s) and exits as soon as the secret is present.
  • The caBundle check then ran once, immediately, with no retry.

But the controller patches caBundle onto rbgs-validating-webhook-configuration asynchronously, after creating the TLS secret. Timestamps from the failed run:

Time (UTC) Event
14:18:32.957 webhook-cert secret check passes (loop exits)
14:18:33.052 caBundle check runs → fails (not patched yet)
14:18:33.162 controller logs patched caBundle on ValidatingWebhookConfiguration

The check ran ~95ms after the secret appeared; the controller finished patching ~110ms after the check. The controller logs confirm the patch succeeded moments later, so the bootstrap is fine — only the verification was too eager.

Fix: wrap the caBundle check in the same retry loop pattern used for the webhook-cert secret (up to 24 × 5s), and dump controller logs on final failure.

Flake 2 — e2e-test-deprecated-disabled job: apicompat update tests hit optimistic-concurrency conflict

The spec rejects a RoleBasedGroupSet update that introduces a deprecated workload type failed:

[FAILED] update introducing a StatefulSet workload should be rejected, got:
  Operation cannot be fulfilled on rolebasedgroupsets... "dep-rbgset-update":
  the object has been modified; please apply your changes to the latest version and try again

The update specs (RBG and RBGSet) did a single Get -> mutate -> Update. The controller writes status on freshly created objects, so its status patch bumps the resourceVersion between the test's Get and Update, surfacing a 409 conflict instead of reaching the validating webhook. containsRejection correctly treats a conflict as "not a rejection", so the spec flakes (it passed on PR #441's run, failed on this branch's run).

Fix: add updateWithConflictRetry, which re-Gets and retries on conflict (up to 10 attempts) and returns the final non-conflict error for the existing rejection assertion; apply it to both update specs in test/e2e/apicompat/specs_test.go.

Both fixes must land together — neither flake is in the other's job, but each PR's CI runs the full e2e suite, so splitting them leaves each PR red on the other's flake.

Test plan

  • go test -c ./test/e2e/apicompat/ compiles; go vet clean; gofmt clean.
  • e2e-test job: verify step passes even when the controller's caBundle patch lands after the TLS secret.
  • e2e-test-deprecated-disabled job: update specs pass even when the controller writes status between Get and Update.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings August 29, 2026 14:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces CI flakiness in the e2e-test workflow by addressing a race where the controller’s cert-manager patches caBundle onto the ValidatingWebhookConfiguration asynchronously after the TLS secret becomes available.

Changes:

  • Wrap the caBundle injection verification in a retry loop (up to 24 attempts with 5s sleep).
  • On final failure, emit the validating webhook configuration YAML and tail controller logs for easier debugging.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The "Verify controller RBAC and webhook-cert bootstrap" step checked the
caBundle on rbgs-validating-webhook-configuration exactly once, immediately
after the webhook-cert secret became present. The controller patches that
caBundle asynchronously after creating the TLS secret, so the one-shot check
races the controller and flakes (see failed run on PR sgl-project#441).

Wrap the caBundle check in the same retry loop used for the webhook-cert
secret (up to 24 x 5s), and dump controller logs on final failure. The
bootstrap itself is fine; only the verification was too eager.

Closes sgl-project#442
@cheyang
cheyang force-pushed the fix/e2e-cabundle-retry branch from 76ad3e5 to 3d2f1ee Compare August 29, 2026 14:27
…flict

The "rejects ... update that introduces a deprecated workload type" specs
(RBG and RBGSet) did a single Get -> mutate -> Update. The controller
writes status on freshly created objects, so its status patch can bump the
resourceVersion between the test's Get and Update, surfacing a 409 conflict
instead of reaching the validating webhook. containsRejection correctly
treats a conflict as "not a rejection", so the spec fails even though the
bootstrap is fine — a flake that passed on PR sgl-project#441's run but failed on sgl-project#443.

Add updateWithConflictRetry, which re-Gets and retries on conflict (up to 10
attempts) and returns the final non-conflict error for the existing
rejection assertion. Apply it to both update specs.

See sgl-project#442
@cheyang cheyang changed the title fix(ci): retry caBundle injection check to avoid cert-manager race fix(e2e): caBundle verify race + apicompat update conflict flake Aug 29, 2026
@diw-zw
diw-zw merged commit c2a0894 into sgl-project:main Aug 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e CI flakiness on v0.8.0 release: caBundle verify race + apicompat update conflict

3 participants