Repository navigation
feat(journaling): route Garnet journals across partitions with a MetaStore - #11501
Draft
ReubenBond wants to merge 15 commits into
Draft
ReubenBond wants to merge 15 commits into
ReubenBond wants to merge 15 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved deadline enforcement, wildcard startup, shutdown connection ownership, and retry-test correctness issues block approval.
4 open findings
What changed in this PR
Extends the experimental Garnet journal provider from #11498 with static multi-partition hosting and MetaStore-backed routing.
Changes:
- Adds validated range maps and in-memory/Azure Blob MetaStores.
- Routes operations and catalog reads by bucket ownership, with endpoint publication and retries.
- Adds routing, metadata, and multi-silo integration coverage.
Automated coverage results were unavailable.
| File | Description |
|---|---|
| test/Extensions/Orleans.Journaling.Garnet.Tests/RangeMetaStoreTests.cs | Tests MetaStore write contracts. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/RangeMapTests.cs | Tests validation and serialization. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/Orleans.Journaling.Garnet.Tests.csproj | Configures provider tests. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetTestRouting.cs | Provides routing helpers. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetPartitionFixture.cs | Manages test partition lifecycle. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalSuiteTests.cs | Adapts shared journaling suites. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalStorageTests.cs | Covers storage behavior. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalStorageOptionsTests.cs | Tests configuration validation. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalRouterTests.cs | Tests ownership and routing retries. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalProtocolTests.cs | Covers protocol and recovery behavior. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalLatencyTests.cs | Measures append performance. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetJournalDurabilityTests.cs | Exercises process-kill recovery. |
| test/Extensions/Orleans.Journaling.Garnet.Tests/GarnetDurableGrainTests.cs | Tests multi-silo journal distribution. |
| test/Extensions/Orleans.Journaling.Garnet.CrashHost/Program.cs | Hosts crash-test partitions. |
| test/Extensions/Orleans.Journaling.Garnet.CrashHost/Orleans.Journaling.Garnet.CrashHost.csproj | Configures crash-test executable. |
| src/Orleans.Journaling/Orleans.Journaling.csproj | Grants provider internal access. |
| src/Garnet/Orleans.Journaling.Garnet/README.md | Documents setup and behavior. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/RangeMapJsonSerializer.cs | Serializes versioned maps. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/RangeMap.cs | Defines maps and invariants. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/PartitionId.cs | Defines validated partition identities. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/IRangeMetaStore.cs | Defines metadata storage contracts. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/InMemoryRangeMetaStore.cs | Implements in-memory conditional writes. |
| src/Garnet/Orleans.Journaling.Garnet/Ranges/AzureBlobRangeMetaStore.cs | Implements durable conditional writes. |
| src/Garnet/Orleans.Journaling.Garnet/Procedures/GarnetJournalProtocol.cs | Adds catalog procedure identification. |
| src/Garnet/Orleans.Journaling.Garnet/Orleans.Journaling.Garnet.csproj | Configures provider packaging and dependencies. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalStorageProvider.cs | Coordinates hosting and catalog access. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalStorageOptionsValidator.cs | Validates provider options. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalStorageOptions.cs | Exposes hosting and routing configuration. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalStorageHostingExtensions.cs | Registers the provider. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalStorage.cs | Sends journal operations through routing. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalRouter.cs | Manages views, connections, and retries. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalPartitionManager.cs | Initializes partitions and publishes endpoints. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalPartitionHost.cs | Hosts durable embedded partitions. |
| src/Garnet/Orleans.Journaling.Garnet/GarnetJournalKeys.cs | Scopes bucket keys by map identity. |
| src/Garnet/DESIGN.md | Records phase-two architecture and decisions. |
| src/api/Garnet/Orleans.Journaling.Garnet/Orleans.Journaling.Garnet.cs | Records generated public APIs. |
| Orleans.slnx | Includes Garnet projects. |
| Directory.Packages.props | Pins the Garnet dependency. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 15, 2026 03:49
004530c to
954df8b
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Crash-safe identity initialization, wildcard endpoint handling, and a MetaStore assertion need correction.
7 open findings
🧠 Review effort: Balanced
Contributor
Code coverage
Report-only conclusion: improved. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Distributed-storage guarantees require human validation, and the deadline regression test needs correction.
1 open finding
7 resolved since last review
🧠 Review effort: Balanced
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 15, 2026 15:40
9524a01 to
f7741eb
Compare
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 15, 2026 17:09
f7741eb to
b9bc405
Compare
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 15, 2026 21:53
b9bc405 to
132f2f8
Compare
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 16, 2026 07:27
132f2f8 to
9820abf
Compare
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 18, 2026 17:51
9820abf to
5dc87e7
Compare
Adds Orleans.Journaling.Garnet, which stores journals in an embedded Garnet partition with synchronous AOF durability (phases 0 and 1 of the range-partitioned design in src/Garnet/DESIGN.md). Each journal operation is one Garnet transaction procedure that checks its bucket's state (the in-band seal used by range migration), applies conditional writes, and recognizes retried mutations by their client-generated ETags. Tests cover the provider contract, the Orleans.Journaling suites, a grain cluster, and process-kill recovery.
…age from the docs inventory
… and bound each attempt Every mutation is now conditioned on a state that never recurs: a content or metadata ETag, or an absence named by its tombstone's ETag. A retry whose ETag has left the eight-entry history can therefore not resurrect a deleted journal or overwrite a later metadata update; it throws GarnetJournalAmbiguousConflictException, an InconsistentStateException. Unconditional metadata updates are conditioned on the ETag the client reads, no-op updates record their ETag, OperationTimeout bounds each attempt's wait, and the bucket count is claimed atomically before any bucket opens.
The API-diff workflow built every source project for net8.0, so net10.0-only projects such as Orleans.Journaling.Garnet failed with NETSDK1005 and kept a stale surface. It now uses net8.0 when targeted and the project's first target framework otherwise.
… in the Garnet durability test SAVE checkpoints while writers append and exceeded the 5 s default async timeout on a Windows CI runner. Each cycle now also waits, bounded, until every writer has an acknowledged append before the random delay and kill.
…Store Add a pluggable range MetaStore (in-memory and Azure Blob, compare-and-swap from V to V+1), host the partitions the range map assigns to each silo, publish their endpoints in the map, and route each journal operation to its bucket's owner, refreshing the view on WrongPartition and retrying unavailable partitions within OperationTimeout. The catalog lists each bucket's index at its owner.
…host ids Bucket keys carry the range map's identity, so a partition of another map reached through a stale endpoint rejects operations. The default HostId is generated once and stored in the data directory instead of using the silo name, which Orleans randomizes. The Azure Blob MetaStore stamps each document with a write id and reports an unknown outcome instead of a wrong one when a later write supersedes an ambiguous upload.
…ialization Address review feedback on the Garnet routing layer: OperationTimeout now bounds connection, command, refresh, and backoff waits without cancelling the shared view refresh; router disposal stops retries and never reopens connections; partitions record their identity before opening buckets and mark initialization complete separately, so interrupted initializations resume only for the same partition and map; silos initialize wildcard-bound partitions through loopback. Compare recovered MetaStore writes by value in tests.
…o routing Port layer 1's retry and initialization changes to the multi-partition router: the router reports attempts whose outcome is unknown through the ambiguous flag while WrongPartition rejections leave it unset, storage passes the flag so conflicts after unknown outcomes surface as GarnetJournalAmbiguousConflictException, and partitions claim their whole identity in one set-if-absent field so concurrent initializations with different maps or bucket counts cannot both succeed. Port layer 1's client tests to the router and partition manager.
…router operations Cover each outcome of a conditional upload which fails its condition with deterministic fault injection: this write's document at the next version reports success, another writer's identical map under a different write id reports failure, and a later version reports an unknown outcome. Router tests apply their watchdog outside the exception assertion, so only an operation's own exception satisfies it.
…eguard A bucket which is open with an incomplete migration and a nonempty local index makes OJINDEX return NeedPull and fails a full catalog enumeration with InvalidOperationException; once the migration completes, the bucket lists its journals.
Disposal marks the router stopped before closing connections, and closing a connection fails commands in flight with cancellation or pipe errors. Operations now report ObjectDisposedException for any failure observed after disposal begins, keeping the original as the inner exception, while cancellation from a connection retired by a view change is retried as an ambiguous connection failure.
ReubenBond
force-pushed
the
rb-feat-journaling-garnet-phase2
branch
from
October 19, 2026 21:06
5dc87e7 to
1d51141
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #11498 (stacked); review only the commits after it.
Problem
Layer 1 runs one Garnet partition that owns every bucket. The range-partitioned design needs many partitions, a MetaStore that is the authority for which partition owns each bucket and where each partition runs, and clients that route by a cached view of that map. That routing has to stay correct when the view is stale, so later phases can move ranges online.
Solution
Phase 2 of
src/Garnet/DESIGN.md. The range map is static: it can express migrations and hosting, but ranges don't move yet.MetaStore (
Orleans.Journaling.Ranges, with no Garnet or Orleans types).IRangeMetaStoreperforms a compare-and-swap from versionVtoV+1on aRangeMap: map identity, bucket count, ranges, migrations, and partition hosting with a published endpoint. Two implementations:InMemoryRangeMetaStore, for tests.AzureBlobRangeMetaStore: one JSON document, written conditionally on its ETag. Each document carries a write id, so after an ambiguous retried upload the store reports the true outcome or "unknown", never a wrong one.RangeMap.Validateenforces the structural invariants.Hosting. Each silo hosts the partitions the map assigns to its
HostId. Each partition gets its own data directory and endpoint, opens its initial buckets, records a descriptor (map identity, partition id, bucket count), and then publishes its endpoint by CAS. IfHostIdis not set, it is generated once and stored in the data directory.Routing.
GarnetJournalRoutermaps JournalId → bucket → owner using the cached view, which it updates in version order, with concurrent refreshes sharing one MetaStore read. It handles failures as follows:WrongPartitionwith a newer bucket version: refresh the view and retry at once.OperationTimeout.Retries are idempotent through the existing recent-ETag mechanism.
Catalog. Each bucket's index is read at its owner with a new
OJINDEXprocedure. It returnsNeedPullfor buckets whose migration is not done, rather than returning a partial listing.Rationale and refinements (recorded in DESIGN.md)
WrongPartitiontells the client whether to refresh or back off.PartitionIdformat, keeping the Azure store in this package (Garnet already depends onAzure.Storage.Blobs), the persisted defaultHostId, and the optional Garnet MetaStore (Undecided).Tests
AzureStoragecategory and pass against Azurite.RangeMap.DurableGrainTestsand checks that journals spread over both hosting silos and are all listed by the catalog.Microsoft Reviewers: Open in CodeFlow