[ReubenBond]

Problem

A starting silo can observe Created -> Dead -> Joining -> Active when another silo joins before it writes its own membership entry. The initial synthetic Created row disappears on refresh, and membership consumers interpret that removal as death. Both ring providers treat local termination as permanent, so this can leave the silo routing reminder registrations away from itself.

Thanks to @nightBaker for the detailed investigation, reproduction, and PR #11581 which identified this race and its reminder-delivery impact. This PR builds on that work and supersedes its proposed implementation. The original contribution is retained in the commit history, and its regression scenario is extended here.

Solution

Keep canonical membership and local lifecycle status separate:

• Initialize membership with an empty MinValue snapshot and local status Created.
• Publish authoritative membership entries to local observers and gossip consumers.
• Derive local Dead status and self-termination when a previously registered silo disappears, while publishing the actual removal.
• Reconcile delayed successful writes and older registration evidence with the current view under the existing publication lock. A newer active status remains active; registration followed by pruning establishes terminal death.

The regression coverage exercises startup notifications, both ring providers' local routing, canonical pre-join cleanup gossip and same-version heartbeat refreshes, removal after registration, and delayed write completion across newer views and same-version pruning.

Rationale

Membership versions describe authoritative cluster state. Keeping an uncommitted local Created row in a published snapshot makes the same version mean different things on different silos. Startup predecessor cleanup can gossip that row before registration; a peer can then reject a same-version authoritative refresh because removing a non-dead row violates the successor rules.

This change addresses the source of the false removal by removing synthetic membership entries, including the existing inferred Dead entry. Local lifecycle reconciliation uses the existing serialized publication boundary and preserves the ring providers' terminal-status contract. All hosts are assumed to run this implementation together.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

🔵 Needs a closer look

Membership-driven self-termination and routing changes warrant final human review, with passing CI and coverage evidence not supplied.

No open issues 🎉

What changed in this PR

This PR separates canonical cluster membership from local lifecycle status, preventing startup refreshes from falsely marking a silo dead and disrupting reminder routing.

Changes:

• Publish authoritative membership snapshots without synthetic local entries.
• Reconcile delayed registration evidence and row removal while preserving terminal local death.
• Document the contracts and add regression coverage for startup routing, gossip, pruning, and delayed writes.

File	Description
test/​Orleans.Core.Tests/​Membership/​MembershipTableManagerTests.cs	Adds canonical-view, routing, and delayed-write regression tests.
src/​Orleans.Runtime/​MembershipService/​MembershipTableManager.cs	Separates published membership from local status and reconciles registration evidence.
src/​Orleans.Runtime/​MembershipService/​IMembershipManager.cs	Documents canonical snapshots and local lifecycle status contracts.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

An in-flight write can overwrite terminal local death before the termination check observes it.

1 open issue

• Prevent Joining from resurrecting an established local Dead state · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Concurrency-sensitive membership publication and self-termination changes require human validation with execution evidence.

1 open issue

• Prevent Joining from resurrecting an established local Dead state

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

An in-flight retry can still overwrite terminal local death and bypass self-termination.

1 open issue

• Prevent Joining from resurrecting an established local Dead state

_{Review effort: Balanced}

[github-actions]

Code coverage

Metric	Pull request	Current main	Variance
Lines	84.00% (120,488 / 143,444)	83.97% (120,451 / 143,437)	+0.0217 pp
Branches	73.83% (36,068 / 48,854)	73.77% (36,036 / 48,846)	+0.0534 pp

Report-only conclusion: improved.

The current-main baseline is commit efffe4681c and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

• Pull request commit: 47ef00f376
• View the pull request test run and coverage artifacts
• View the trusted coverage reporting run

[Copilot]

🔵 Needs a closer look

Asynchronous membership reconciliation and cleanup retention affect cluster safety and warrant final human review despite no confirmed blocking findings.

No open issues 🎉

1 resolved since last review

• Prevent Joining from resurrecting an established local Dead state

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Distributed membership ordering, terminal lifecycle reconciliation, and cleanup retention warrant final maintainer review despite no confirmed blocking findings.

No open issues 🎉

_{Review effort: Balanced}