Repository navigation
feat(membership): retain local death knowledge for absent silos - #11589
Merged
ReubenBond merged 4 commits intoOct 27, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Zero-age startup, retained successor evidence, and older snapshot deserialization have unresolved correctness issues.
3 open issues
What changed in this PR
This PR builds on #11588’s canonical membership views to retain death knowledge after silo rows are pruned, supporting reliable client routing and durable-job adoption.
Changes:
- Adds configurable generation retirement and retained death records without changing persisted membership schemas.
- Publishes same-version status updates and incorporates them into routing and adoption decisions.
- Adds regression tests, configuration validation, API metadata, and documentation.
Validation was limited to source inspection; test and coverage results were not verified.
| File | Description |
|---|---|
test/Orleans.Runtime.Internal.Tests/ClusterMembershipSnapshotTests.cs |
Tests status classification, pruning, and serialization. |
test/Orleans.DurableJobs.Tests/DurableJobs/JournaledJobShardRetirementTests.cs |
Tests adoption accounting after owner pruning. |
test/Orleans.DurableJobs.Tests/DurableJobs/JournaledJobShardManagerTests.cs |
Adds membership removal support to the test helper. |
test/Orleans.Core.Tests/Membership/MembershipTableSnapshotTests.cs |
Tests death retention, merging, and cutoff boundaries. |
test/Orleans.Core.Tests/Membership/MembershipTableManagerTests.cs |
Tests canonical startup views and delayed registration reconciliation. |
test/Orleans.Core.Tests/Membership/MembershipRetirementTests.cs |
Tests retirement publication with controlled clocks. |
test/Orleans.Core.Tests/Membership/ClusterMembershipOptionsTests.cs |
Tests retirement defaults and validation. |
test/Orleans.Core.Tests/Directory/MockClusterMembershipService.cs |
Supports same-version snapshot publication in tests. |
test/Orleans.Core.Tests/Directory/ClientDirectoryTests.cs |
Tests stale-route removal and delayed-update rejection. |
src/Orleans.Runtime/MembershipService/SiloStatusOracle.cs |
Returns local lifecycle status separately from membership rows. |
src/Orleans.Runtime/MembershipService/MembershipTableSnapshotExtensions.cs |
Carries retirement knowledge into cluster snapshots. |
src/Orleans.Runtime/MembershipService/MembershipTableManager.cs |
Applies retirement during publication and reconciles local status. |
src/Orleans.Runtime/MembershipService/IClusterMembershipService.cs |
Documents same-version membership updates. |
src/Orleans.Runtime/MembershipService/ClusterMembershipSnapshot.cs |
Adds retained death knowledge and successor checks. |
src/Orleans.Runtime/MembershipService/ClusterMembershipService.cs |
Publishes and exposes same-version status changes. |
src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs |
Refreshes routes using snapshot identity and retained status. |
src/Orleans.Runtime/Configuration/Validators/SiloClusteringValidator.cs |
Rejects negative retirement ages. |
src/Orleans.DurableJobs/JournaledJobShardManager.cs |
Uses recorded membership versions when classifying previous owners. |
src/Orleans.Core/Runtime/MembershipTableSnapshot.cs |
Retains and compacts death evidence separately from canonical rows. |
src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs |
Adds the nullable retirement-age option. |
src/Orleans.Core.Abstractions/IDs/SiloAddress.cs |
Calculates generation retirement floors. |
src/api/Orleans.Core/Orleans.Core.cs |
Exposes the new option in the generated API surface. |
docs/site/src/content/docs/implementation/cluster-management.md |
Explains retirement behavior, retention, and operational limits. |
Review effort: Balanced
Keep canonical membership rows separate from a local publication-time generation cutoff and recent death records. Default retirement age to one hour with a nullable opt-out, and preserve version-aware and fresh-validation death evidence. Wire same-version client-route invalidation and durable-job adoption accounting. Document the accepted prolonged stalled-write race and the deferred central membership admission fence.
Disable generation retirement in health-monitor and membership-agent fixtures whose manually assigned generations are unrelated to their clocks. Dedicated retirement tests continue to exercise the enabled cutoff.
ReubenBond
force-pushed
the
rb-membership-retirement-watermark
branch
from
October 27, 2026 15:21
d4bf214 to
964e320
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Older gossip payloads can fail to deserialize safely, and pruning a successor can still erase death knowledge.
5 open issues
Review effort: Balanced
Keep an unregistered local silo Created when only the generation cutoff implies death, including zero-age retirement. Preserve explicit death and delayed registration/removal evidence even after compaction, and reject equal-version omissions as local death evidence. Scan retained successor identities so pruning cannot revive predecessor routes. Isolate synthetic-generation directory contract fixtures from age-based retirement and cover the transitions with deterministic regressions.
Expose IReadOnlySet instead of ImmutableHashSet, build updates in a private HashSet, and freeze published history to preserve snapshot ownership. Share unchanged frozen histories across updates and cluster projections. Task-scoped lookup measurements favor FrozenSet over ImmutableHashSet, with HashSet offering similar lookups but no physical immutability. Cover mutable inputs, set reuse, compaction, and interface-typed serialization on both target frameworks.
Contributor
Code coverage
Report-only conclusion: mixed. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
This was referenced Oct 27, 2026
ReubenBond
added a commit
to ReubenBond/orleans
that referenced
this pull request
Oct 27, 2026
dotnet#11589) (cherry picked from commit 1643d2d)
ReubenBond
added a commit
to ReubenBond/orleans
that referenced
this pull request
Oct 27, 2026
dotnet#11589) (cherry picked from commit 1643d2d)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This follow-up builds on the canonical membership-view fixes merged in #11588 and is rebased onto
main. The original missing-silo diagnosis was reported by @nightBaker in #11581.Problem
Once membership cleanup prunes a dead silo's row, ordinary status lookups can return
Noneand lose the evidence of death. Delayed client-route updates can then retain or reintroduce stale routes, and durable-job adoption can treat a pruned owner as a graceful release and reset its adoption count.Solution
Keep canonical membership rows authoritative and retain local status knowledge separately: present rows supply their actual status; absent identities use a fixed generation cutoff, compact recent-death records, and existing successor/version-aware evidence.
ClusterMembershipOptions.SiloGenerationRetirementAgeis a nullableTimeSpan, defaulting to one hour. Each peer establishes its cutoff from its local UTC clock when publishing a view. The cutoff advances monotonically, and fixed snapshots keep fixed status answers.nulldisables age-based retirement, negative values are configuration errors, and zero is valid. Present long-running members retain their actual status regardless of generation age.Observed deaths remain individually recorded until the cutoff covers their generations. With retirement enabled and normal timestamp-derived generations, history follows the recent generation window rather than cluster lifetime; it is a time-window bound, not a byte cap. With retirement disabled, observed death records remain for the observer's lifetime. A new observer can use versioned prior-registration evidence or existing fresh validation to resolve recent absent identities before the cutoff.
Publish same-version status-knowledge updates, refresh client routing on those snapshots, remove retired peers even after their rows are gone, and use the recorded membership version for durable-job adoption accounting. Local pre-registration
Createdstatus stays separate from canonical rows. Storage providers and persisted membership schemas remain unchanged.Deferred strengthening
Microsoft Reviewers: Open in CodeFlow