RFC: Deprecate and remove RecreateRBGOnPodRestart restart policy #339

Summary

This RFC proposes deprecating and removing the RecreateRBGOnPodRestart restart policy and its associated Pod Controller from the RBG project.

Motivation

The RecreateRBGOnPodRestart policy was originally designed to recreate all workloads in an RBG when any pod experiences a restart (container crash, pod deletion, etc.). In practice, this policy has no valid production use case — operators overwhelmingly prefer finer-grained restart strategies like RecreateRoleInstanceOnPodRestart (which recreates only the affected instance) or None (which relies on the workload controller's native replacement logic).

Meanwhile, the implementation introduces significant architectural complexity and operational risk:

1. Dual-controller pod management creates state races

The RecreateRBGOnPodRestart policy requires a dedicated Pod Controller that watches all pod events and independently manages RBG status conditions (RestartInProgress). This creates a race condition between the Pod Controller and the RBG Controller:

• Both controllers write to rbg.status.conditions using Server-Side Apply with Force=true
• The RBG Controller can overwrite the Pod Controller's condition updates, causing RestartInProgress=True to become permanently stuck
• This manifests as flaky e2e tests that are difficult to reproduce and fix reliably

2. Increased control plane pressure

The Pod Controller watches all pods labeled with RBG ownership and triggers full RBG reconciliation on any pod event (restarts, deletions, status changes). This amplifies the number of API server requests and reconciliation cycles, particularly in large clusters with many RBG-managed workloads.

3. Code complexity with no offsetting value

The feature adds:

• An entire controller (pod_controller.go) with its own event filtering, RBAC, and reconciliation logic
• A RecreateWorkload method on every WorkloadReconciler implementation (Deployment, StatefulSet, LeaderWorkerSet, RoleInstanceSet)
• A dedicated status condition type (RestartInProgress) with its own lifecycle management
• Special-case field manager handling to attempt (unsuccessfully) to avoid SSA conflicts

All of this complexity serves a policy that no known deployment uses in production.

Proposal

1. Remove the RecreateRBGOnPodRestart enum value from the RestartPolicy field in both v1alpha1 and v1alpha2 API versions
2. Remove the Pod Controller entirely
3. Remove the RestartInProgress condition type
4. Remove the RecreateWorkload method from the WorkloadReconciler interface
5. Update CRD validation to only allow {None, RecreateRoleInstanceOnPodRestart}

Migration Path

• Existing RBGs with restartPolicy: RecreateRBGOnPodRestart should be updated to either:
  • RecreateRoleInstanceOnPodRestart — recreates the affected role instance (recommended for most distributed workloads)
  • None — relies on the workload controller's native pod replacement (recommended for stateless workloads)
• Since this is an alpha API, breaking changes are acceptable per Kubernetes API conventions

Alternatives Considered

Fix the race condition with separate field managers: We attempted to use distinct SSA field managers (rbg vs rbg-pod-controller) to allow independent ownership of different conditions. While this resolves the immediate conflict, it doesn't address the fundamental architectural issue of dual-controller pod management or the unnecessary control plane pressure.

References

• Related flaky test investigation: RestartInProgress condition stuck at True indefinitely due to SSA race between controllers
• Kubernetes SSA documentation on field ownership and Force=true semantics