Skip to content

e2e CI flakiness on v0.8.0 release: caBundle verify race + apicompat update conflict #442

Closed

Description

@cheyang

Summary

The v0.8.0 release PR (#441) exposed two independent e2e CI flakes. Both are pre-existing races, not v0.8.0 regressions.

Failed run: https://github.com/sgl-project/rbg/actions/runs/33257029727/job/99112489896?pr=441

Flake 1 — e2e-test: caBundle verify races the cert-manager patch

The "Verify controller RBAC and webhook-cert bootstrap" step in .github/workflows/e2e-test.yml failed with:

::error::caBundle not injected into every webhook of rbgs-validating-webhook-configuration

The step has two checks:

  1. webhook-cert secret — wrapped in a retry loop (for i in $(seq 1 24) ... sleep 5), so it waits up to ~120s.
  2. caBundle on ValidatingWebhookConfiguration — runs once, immediately after the secret check passes, with no retry.

The controller patches caBundle onto rbgs-validating-webhook-configuration asynchronously, after creating the TLS secret. Timestamps from the failed run:

Time (UTC) Event
14:18:32.957 webhook-cert secret check passes (loop exits)
14:18:33.052 caBundle check runs → fails (not yet patched)
14:18:33.162 controller logs patched caBundle on ValidatingWebhookConfiguration

The caBundle check ran ~95ms after the secret became present, but the controller didn't finish patching the VWC until ~110ms after the check. The controller logs confirm the patch succeeded moments later, so the bootstrap is fine — only the verification was too eager.

Fix: mirror the retry loop used for the webhook-cert secret around the caBundle check.

Flake 2 — e2e-test-deprecated-disabled: apicompat update tests hit optimistic-concurrency conflict

The spec rejects a RoleBasedGroupSet update that introduces a deprecated workload type failed:

[FAILED] update introducing a StatefulSet workload should be rejected, got:
  Operation cannot be fulfilled on rolebasedgroupsets... "dep-rbgset-update":
  the object has been modified; please apply your changes to the latest version and try again

The update specs (RBG and RBGSet in test/e2e/apicompat/specs_test.go) do a single Get -> mutate -> Update. The controller writes status on freshly created objects, so its status patch bumps the resourceVersion between the test's Get and Update, surfacing a 409 conflict instead of reaching the validating webhook. containsRejection correctly treats a conflict as "not a rejection", so the spec flakes (it passed on #441's run).

Fix: retry the Get->mutate->Update on conflict and return the final non-conflict error for the existing rejection assertion.

Fix

PR #443 addresses both flakes.

Activity

  1. added 2 commits that reference this issue on Aug 29, 2026
    3d2f1ee
    7595aa4
  2. changed the title [-]e2e-test job flake: caBundle not injected check races with controller cert-manager patch[/-] [+]e2e CI flakiness on v0.8.0 release: caBundle verify race + apicompat update conflict[/+] on Aug 29, 2026
  3. added a commit that references this issue on Aug 30, 2026
    c2a0894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions