[divyvasal]

Motivation

Fixes #42465. With DSPARK in greedy mode (folded accept) and the overlap scheduler on, TP ranks can commit different token counts for the same request.

DsparkVerifyEpilogue.read_accept() returns views of fixed buffers:

• commit_lens_buf, out_tokens_buf, correct_len_buf, bonus_buf, cap_trim_lens_buf, new_seq_lens_buf;
• the folded accept kernels write them inside the target-verify CUDA graph;
• the next replay overwrites them.

Under event_loop_overlap:

• the result device-to-host copy (accept_lens / next_token_ids) runs on copy_stream;
• forward_stream never waits on that copy;
• the loop launches step N+1 before it consumes step N's copy.

A copy that runs late therefore reads step N+1's values. The race resolves independently on each rank, so ranks disagree on kv_committed_len.

With SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1:

[TP0/1/2] Found rank divergence ... cache_finished_req ... owned_kv_len=531526
[TP3]     Found rank divergence ... cache_finished_req ... owned_kv_len=531528

With HiCache (write_through), the divergent inserts become divergent backups. The ranks end up one collective apart and the replica hangs, as described in #42465.

Why HiCache triggers it: HiCache's D2H write-through traffic delays the small result copy by a different amount on each rank. Under that traffic the race fires within minutes.

Without HiCache: the race is still present, but we did not observe it in ~21M tokens. If it does fire, it can also return the next step's next_token_ids on rank 0.

Modifications

• read_accept() returns .clone() of each buffer.
• The clones are made on the forward stream right after the verify replay, so they capture step N's values.
• The allocator owns them, so async_d2h's record_stream keeps them alive until the copy has run.
• Cost: six tiny device-to-device copies per verify step.
• The overlap scheduler stays on. --disable-overlap-schedule also avoids the bug, but cost us 14-20% per-request decode throughput.

Accuracy / validation

Setup: DeepSeek-V4.1-Flash (MXFP4), TP4/EP4 on 4x B300, two replicas per node, nightly 4cbf290f plus this change.

Flags:

--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --schedule-policy shortest-prefill-first
--chunked-prefill-size 8192 --enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through
--mem-fraction-static 0.60

The checker was on. Workload: greedy (temperature 0) unique ~540K-token prompts, max_tokens 4; 1 in flight on one replica, 8 on the other.

Build	Result
without this change	both replicas hung (one within ~1 min, one within ~19 min); earlier runs logged the owned_kv_len divergence above within 8-12 min
with this change, run 1 + run 2	43.1M tokens per replica, 0 rank divergence, 0 hangs/restarts

Controls on the same hardware and workload, ~21M tokens each, checker on:

Config	Rank divergence
HiCache without DSPARK	none
DSPARK without HiCache	none
HiCache + DSPARK with --disable-overlap-schedule	none

So the bug needs DSPARK, HiCache and the overlap scheduler together.

New CPU unit test: test/registered/unit/spec/test_dspark_read_accept_no_alias.py.

• It asserts that the returned tensors do not share storage with the buffers.
• It asserts that they keep their values after the buffers are overwritten.
• It fails on main and passes with this change.

Lint: isort / ruff format / ruff check --select=F401,F821,UP037 / codespell all clean.

Checklist

• Unit test added
• Formatted with the repo's pre-commit tools

---

CI States

Latest PR Test (Base): ❌ Run #40133351715
Latest PR Test (Extra): ❌ Run #40133351725
Latest PR Test (AMD ROCm 10): ❌ Run #40133351776

[divyvasal]

@hnyls2002 @ch-wan could one of you take a look and add run-ci? This fixes the TP-rank deadlock in #42465: read_accept() returned views of buffers that the next verify replay overwrites, so under the overlap scheduler a late result copy could read the next step's accept lengths on some ranks only. The change returns copies instead; validated at 43M tokens per replica with 0 rank divergence (details in the description).

[shanemort1982]

Heads-up before this lands: on current main, DsparkVerifyEpilogue._accept writes the in-graph results through build_accept_outs(..., out=self.read_accept(bs)) (dspark_verify.py ~L799 at 62cee1b). With this PR's change, read_accept returns clones, so those writes go into throwaway tensors, the persistent buffers stay at zero, and the folded path reads accept_len 0 on every step. We hit exactly that porting this PR onto a recent nightly: num_correct_drafts comes out as -1 and the scheduler dies with an IndexError in update_spec_correct_drafts_histogram on the first GPU decode step.

The PR head still applies against main's layout because it predates the build_accept_outs(out=...) refactor (the head branch fills the buffers with explicit copy_() calls), so it merges cleanly but breaks at runtime.

What worked for us: keep a views accessor as the write target and clone only for the reader.

def _accept_buffers(self, bs):   # views; out= target inside the graph
    return AcceptOuts(correct_len=self.correct_len_buf[:bs], ...)

def read_accept(self, bs):       # clones, as this PR intends
    b = self._accept_buffers(bs)
    return AcceptOuts(correct_len=b.correct_len.clone(), ...)

and _accept passes out=self._accept_buffers(bs). A CPU-only test that writes through _accept_buffers and asserts read_accept returns those values (and keeps them after the buffers are overwritten) catches the regression; the existing alias-only tests pass on the broken form.

[divyvasal]

@shanemort1982 thanks for catching this, and for the clear writeup. You're right: since the build_accept_outs(out=...) refactor, the in-graph writes were landing in the clones. sampling_accept_commit_lens in the sampling accept graph had the same problem.

I rebased onto main and added an accept_buffers(bs) views accessor that both writers now use as the out= target, while read_accept keeps returning copies, so the #42465 fix stays as it was. I also added the CPU test you described. It fails on the clone-as-target form, where the old alias-only tests still pass, and the existing accept-tail parity tests on main catch the break too.

[github-actions]

Required checks failed on 155fa07; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

[0xtoward]

Hi, we hit the same aliasing from a different direction, in case it helps the review.

On 0.5.22 with --speculative-algorithm DSPARK --speculative-dspark-block-size 7 --speculative-dspark-sps-table-path

and SGLANG_RAGGED_VERIFY_MODE=compact, the synchronous loop (--disable-overlap-schedule) loses acceptance: 2.42 vs 3.09 under the overlap scheduler and 3.12 eager, on the same 80 MT-Bench prompts (Qwen3-8B-based target, greedy), while the outputs stay byte-identical.

Mechanism: the sync scheduler adopts GenerationBatchResult.new_seq_lens (a view of new_seq_lens_buf) as the next batch’s seq_lens; the next replay’s in-graph accept rewrites it right after the verify, so inject_ragged computes the draft-KV injection positions from post-commit lengths (positions [13..20] where [10..17] were correct, slots right, RoPE phase wrong). Static verify isn’t affected because its injection uses the window’s precomputed positions; eager returns fresh tensors; overlap keeps its own FutureMap buffer. Returning copies from the folded accept, as this PR does, brings the sync loop back to 3.09.

[divyvasal]

Thanks @0xtoward, this is really helpful! It's the same aliasing showing up in a sneakier way: the outputs stay identical, so you'd only notice the drop in acceptance. That lines up with the mechanism too. Anything that keeps a read_accept result around past the next replay ends up reading the next step's values.

Heads up that I rebased yesterday (now at 155fa07c). On current main the in-graph accept writes its results through out=, so the PR now gives the writers a separate accept_buffers(bs) that returns views, and read_accept still hands out copies. If you have time to rerun your sync-loop MT-Bench check on the new head, I'd love to know it still gets you back to ~3.09.

@hnyls2002 @ch-wan this now covers two issues: the TP-rank deadlock in #42465 and this acceptance loss in the sync loop. Would one of you mind taking a look and adding run-ci?