⭐ Must Read: sgl project/sglang/pull/

DarkSharpness · 2026-07-06T15:43:37Z

(generated by claude)

Summary

Integrates the DeepSeek-V4 top-k v2 JIT kernel (topk_transform_512_v2, runtime k≤2048, added in #26788) into the shared DeepSeek-V3.2 / GLM DSA sparse-attention path, and removes the page_size=1 page-table overhead it previously required. Python/Triton only — the JIT kernel file is unchanged. Three incremental commits:

1. Fold the page-table into fused top-k v2 (decode).
The legacy top-k transform expanded to a page_size=1 page table. The v2 kernel supports arbitrary page size, so _try_topk_transform_v2_paged (in dsa_topk_backend.py) feeds it the compact page_size=64 real_page_table (the indexer's own block table) directly. Engaged behind the existing SGLANG_OPT_USE_TOPK_V2 flag for the decode-shaped PAGED, fp32, 1-row/batch case; everything else (prefill RAGGED, multi-query, non-fp32) falls through to the legacy path unchanged. A one-time INFO log reports engagement for routing observability.

2. Preprocess the v2 plan once per forward.
The cluster-threshold plan (plan_topk_v2) is now computed once per forward batch into DSAMetadata.topk_v2_plan (eager + capture), refreshed in-place with .copy_() on CUDA-graph replay, and reused across layers instead of recomputed per top-k call.

3. Drop the wide page_size=1 page table in fused decode CUDA graphs.
The decode CUDA graph statically allocated a [max_num_tokens, max_ctx_len] page_size=1 table. With fused top-k (default) it is dead weight — attention reads topk_indices, the indexer reads the compact real_page_table. The new dsa_drop_wide_page_table gate (cuda ∧ ¬hip ∧ page_size>1 ∧ no-hisparse ∧ no-spec ∧ fuse_topk) skips the wide allocation and materializes only the compact real_page_table, filled straight from req_to_token by the fused metadata triton kernels (their page_table_1 write is made optional). At 160K context this shrinks the static table 64× (2560 vs 163840 int32 cols/row). Non-fused / hisparse / HIP-indexer / spec (MTP precompute, target-verify, draft-extend) paths keep the wide table and are untouched.

Verification

B200 unit: the three fused metadata triton kernels produce byte-identical real_page_table with page_table_1=None vs the wide-table path.
B200 wiring: the drop-path decode-graph real_page_table is byte-identical (same shape + values) to the legacy derive-from-wide result, verified up to 160K context.
v2 correctness: v2 selection ≡ v1 ≡ torch.topk and the fold yields identical physical KV slots to page_size=1, through the real DSATopKBackend.topk_transform, up to 1M context; CUDA-graph capture/replay safe.
E2E: DeepSeek-V3.2 GSM8K (TP8) — validated via the registered 8-gpu-h200 test.

Compatibility

GLM DSA (index_topk) shares this exact code path, so it is covered without extra code. All new behavior is gated; default configs on non-DSA / non-CUDA / spec paths are byte-unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

CI States

Latest PR Test (Base): ❌ Run #28817867682
Latest PR Test (Extra): ❌ Run #28817866983

…expansion The DeepSeek-V3.2 / GLM DSA sparse-attention decode path selects the per-row top-k of the indexer logits and maps the selected positions through a page table. The legacy fused path (fast_topk_transform_fused / flashinfer top_k_page_table_transform) consumes a page_size=1 page table (attn_metadata.page_table_1) that is `max_seqlen_k` wide per row -- a large buffer that must be captured in the prefill & decode CUDA graphs. The DeepSeek-V4 top-k v2 JIT kernel (topk_transform_512_v2) fuses selection + transform and consumes an arbitrary power-of-two page size, so it can read the indexer's own compact `real_page_table` (page_size = pool page size, typically 64) directly and fold the transform: slot = real_page_table[row, p // page_size] * page_size + (p % page_size) This is arithmetically identical to gathering the page_size=1 table, so the selected physical KV slots are unchanged, while the page-table buffer the transform reads is `page_size`x smaller (64x for the common case). Add `_try_topk_transform_v2_paged`, engaged behind the existing `SGLANG_OPT_USE_TOPK_V2` flag (default on) only for the decode-shaped PAGED case the kernel is defined for (no ragged row_starts, no batch_idx_list remap, one fp32 score row per batch aligned 1:1 with the page table, 0<topk<=2048). Every other case (prefill RAGGED / multi-query PAGED / non-fp32) returns None and falls through to the existing implementation unchanged. Shared by DeepSeek-V3.2 and GLM DSA (both route through DSATopKBackend.topk_transform). Verified on B200: the folded path (real_page_table, ps=64) yields the IDENTICAL physical KV slots as the same kernel run with page_table_1 (ps=1), through the real DSATopKBackend.topk_transform entrypoint, across decode shapes up to 1M context; v2 selection matches v1/torch; helper is CUDA-graph capture/replay safe. Full-model DeepSeek-V3.2 GSM8K e2e pending (671B model, run via CI test/registered/8-gpu-models/test_deepseek_v32_indexcache.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

gemini-code-assist · 2026-07-06T15:43:41Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

DarkSharpness · 2026-07-06T15:43:54Z

(generated by claude)

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

github-actions · 2026-07-06T15:44:19Z

Results for /rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/8-gpu-models/test_deepseek_v32_indexcache.py

DarkSharpness · 2026-07-06T16:05:52Z

(generated by claude)

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

DarkSharpness · 2026-07-06T16:14:44Z

(generated by claude)

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

DarkSharpness · 2026-07-06T16:23:03Z

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

github-actions · 2026-07-06T16:23:28Z

Results for /rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/8-gpu-models/test_deepseek_v32_indexcache.py

Previously `_try_topk_transform_v2_paged` called `plan_topk_v2(lengths)` on every layer's top-k transform. The plan (cluster-threshold metadata) depends only on the decode seq lengths, which are identical across all layers of a forward, so compute it once. Mirror the `paged_mqa_schedule_metadata` lifecycle: a new `DSAMetadata. topk_v2_plan` field computed once at eager metadata build and at CUDA-graph capture, and refreshed in-place (`copy_`, preserving the captured data_ptr) on graph replay via `_refresh_topk_v2_plan`. The refresh self-gates to the decode shape (one score row per batch) the fold engages on, and to `SGLANG_OPT_USE_TOPK_V2`. The helper now consumes `attn_metadata.topk_v2_plan` and only falls back to an inline `plan_topk_v2` when it is absent / shaped for a different batch. Verified on B200: through the real DSATopKBackend.topk_transform entrypoint, both the precomputed-plan reuse and the inline fallback yield the identical (exact) folded physical KV slots as the page_size=1 path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

The decode-family CUDA graph statically allocated a [max_num_tokens, max_ctx_len] page_size=1 table (req_to_token width). With fused top-k (default) it is dead weight: attention reads topk_indices and the indexer reads the compact page_size=64 real_page_table. Only non-fused, hisparse, HIP-indexer, and spec (MTP precompute / target-verify / draft-extend) paths consume it. Gate `dsa_drop_wide_page_table` (cuda ∧ ¬hip ∧ page_size>1 ∧ no-hisparse ∧ no-spec ∧ fuse_topk) skips allocating the wide table and instead materializes only the compact real_page_table, filled straight from req_to_token by the fused metadata triton kernels (page_table_1 write made optional). At 160K ctx this shrinks the static table 64x (2560 vs 163840 int32 cols/row). Verified on B200: the three fused metadata kernels produce byte-identical real_page_table with page_table_1=None, and the decode-graph wiring yields a real_page_table byte-identical (same shape+values) to the legacy derive-from-wide-table result up to 160K context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

DarkSharpness · 2026-07-06T17:22:56Z

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

github-actions · 2026-07-06T17:23:22Z

Results for /rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/8-gpu-models/test_deepseek_v32_indexcache.py

The [DSA] fold-engaged INFO log was added only to prove routing in CI; that is confirmed, so drop it (and the now-unused logging import/global). Also drops a redundant forward-ref quote on a type annotation (ruff). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

DarkSharpness · 2026-07-06T17:39:55Z

/rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py

github-actions · 2026-07-06T17:40:21Z

Results for /rerun-test test/registered/8-gpu-models/test_deepseek_v32_indexcache.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/8-gpu-models/test_deepseek_v32_indexcache.py

…ernels Add a docstring to fused_dsa_decode_metadata describing the optional page_table_1 (pass None to write only the compact real_page_table; requires real_page_size>1), and comments at the three kernel store guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

Reworks the fused decode top-k so correctness no longer depends on a soft try/None fallback that could deref a dropped page_table_1. - topk_transform now dispatches decode-shaped PAGED to the v2 kernel as a deterministic decision on the work shape (SGLANG_OPT_USE_TOPK_V2 + PAGED + no row_starts/batch_idx_list + topk<=2048 + 1-row/batch). Once dispatched it commits: no falling back to the legacy page_size=1 path (which, under the page-table fold, may have no page_size=1 table). The legacy branch asserts page_table_1 is present. - _try_topk_transform_v2_paged -> _topk_transform_v2_paged: its preconditions are now invariants that assert (raise) rather than return None. The DeepGEMM indexer emits contiguous fp32 16B-aligned scores, so drop the pad-to-4; the plan is preprocessed per forward, so drop the inline recompute and assert it is present (a missing plan is a caller bug, not something to paper over). - dsa_drop_wide_page_table now also requires SGLANG_OPT_USE_TOPK_V2 and dsa_index_topk<=2048, matching exactly when decode routes to v2 -- so the page_size=1 table is dropped only when nothing reads it. Verified on B200: folded selection matches a torch top-k reference for topk=512/2048, and a missing/mismatched plan raises. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

DarkSharpness · 2026-07-06T18:41:20Z

/rerun-test test/registered/models_e2e/test_dsa_glm52_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_dp_mtp.py test/registered/models_e2e/test_dsa_glm52_hisparse.py

github-actions · 2026-07-06T18:41:45Z

Results for /rerun-test test/registered/models_e2e/test_dsa_glm52_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_dp_mtp.py test/registered/models_e2e/test_dsa_glm52_hisparse.py:

🚀 8-gpu-h200 (3 tests): ❌ View workflow run

cd test/ && python3 registered/models_e2e/test_dsa_glm52_tp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_dp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_hisparse.py

DarkSharpness · 2026-07-06T18:48:29Z

/rerun-test test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py

github-actions · 2026-07-06T18:48:52Z

Results for /rerun-test test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py:

🚀 4-gpu-b200 (2 tests): ❌ View workflow run

cd test/ && python3 registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py

….2 MTP crash) GLM 5.2 MTP core-dumped with CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS in topk_small_batch_kernel<true> (our v2 kernel). It reproduces only for the spec verify / draft-extend path: small effective batch (bs * num_draft_tokens) plus the wide CUDA-graph score buffer (width = max context) makes the host pick the small-batch cluster launch, and that path faults under the multi-query MTP graphs. Non-spec decode (large batch -> main kernel) is unaffected and is validated at GSM8K 0.955; the v2 fold is only e2e-validated for single-query decode. The kernel is robust to NaN/Inf/large-score/wide-buffer/large-seqlen in isolation (verified on B200), so the fault is a CUDA-graph/PDL/cluster-launch interaction not reproducible out-of-graph. Temporary fix: gate v2 dispatch off for target-verify / draft-extend (allow_topk_v2 on DSAIndexerMetadata, set from the forward mode). Those fall back to the legacy page_size=1 transform, which is safe -- the page-table fold already excludes spec, so page_table_1 is present. Decode/prefill are unchanged. TODO(dsa-topk-v2): re-enable for MTP once the small-batch kernel is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQgtB3k9vngYs9MyAu7BSo

DarkSharpness · 2026-07-06T19:34:52Z

/rerun-test test/registered/models_e2e/test_dsa_glm52_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_dp_mtp.py test/registered/models_e2e/test_dsa_glm52_hisparse.py test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py

github-actions · 2026-07-06T19:38:52Z

Results for /rerun-test test/registered/models_e2e/test_dsa_glm52_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_dp_mtp.py test/registered/models_e2e/test_dsa_glm52_hisparse.py test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py test/registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py:

🚀 8-gpu-h200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_dsa_glm52_tp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_dp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_hisparse.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_nvfp4_dp_mtp.py

The fused v2 PAGED dispatch (from #30274) asserts that DSAMetadata carries a preprocessed topk_v2_plan instead of silently recomputing it per layer. The equivalence test builds DSAMetadata by hand without the plan, so test_topk_fused_backends_equivalence started failing on trunk for every PR (e.g. #30275's base-b-test-1-gpu-large job). Build the plan the same way init_forward_metadata / _build_forward_metadata_cuda_graph do; the sgl-kernel-vs-flashinfer equivalence check now actually covers the v2 fused path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…expansion (sgl-project#30274)

DarkSharpness requested review from 1am9trash, Fridge003, HaiShaw, YAMY1234, hubertlu-tw, kkHuang-amd and rainj-me as code owners July 6, 2026 15:43

DarkSharpness requested review from Qiaolin-Yu, hebiao064, ispobock and merrymercy as code owners July 6, 2026 16:05

DarkSharpness force-pushed the dsa-topk-v2 branch from 6a09f9f to 752e4c0 Compare July 6, 2026 16:14

root and others added 2 commits July 6, 2026 09:40

DarkSharpness force-pushed the dsa-topk-v2 branch from 752e4c0 to 7179694 Compare July 6, 2026 17:21

root and others added 2 commits July 6, 2026 10:43

amd-bot mentioned this pull request Jul 7, 2026

[CI Monitor] Daily Report - 2026-07-07 bingxche/sglang-ci-bot#128

Open

Fridge003 approved these changes Jul 7, 2026

View reviewed changes

Fridge003 merged commit be70bfb into sgl-project:main Jul 7, 2026
93 of 107 checks passed

DarkSharpness mentioned this pull request Jul 7, 2026

[DSA] Re-enable fused top-k v2 for MTP: clamp padded-row seq_lens to >= 0 #30378

Merged

mmangkad mentioned this pull request Jul 7, 2026

[CI] Fix DSA top-k v2 test metadata #30383

Closed

DarkSharpness mentioned this pull request Jul 7, 2026

Fix DSV4 prefill large Triton recompilation idle across context lengths #30255

Merged

fanxingran mentioned this pull request Jul 8, 2026

[AMD] Disable DSA fused top-k v2 on ROCm for GLM-5.x / DeepSeek-V3.2 #30506

Merged

5 tasks

amd-bot mentioned this pull request Jul 8, 2026

[CI Monitor] Daily Report - 2026-07-08 bingxche/sglang-ci-bot#129

Open

yctseng0211 mentioned this pull request Jul 15, 2026

[AMD] DeepSeek-V3.2: disable top-k v2 on rocm (unblock stage-c AMD PR Test) #31291

Closed

5 tasks

This was referenced Jul 17, 2026

[CI Monitor] Daily Report - 2026-07-17 bingxche/sglang-ci-bot#138

Open

[CI Monitor] Daily Report - 2026-07-18 bingxche/sglang-ci-bot#139

Open

JustinTong0323 mentioned this pull request Aug 4, 2026

[Bug] RuntimeError in DSA attention backend during EAGLE verify when sequence approaches context-length (tensor size mismatch 614406 vs 614408) #30570

Closed

5 tasks done

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[DSA] Fold page-table into fused top-k v2 (decode): drop page_size=1 …

c2f099c

…expansion (sgl-project#30274)

hsthe29 mentioned this pull request Oct 2, 2026

[Spec] Keep the speculative lookahead within context_len at the length cap #42244

Open

4 of 5 tasks

Conversation

DarkSharpness commented Jul 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Compatibility

CI States

Uh oh!

gemini-code-assist Bot commented Jul 6, 2026

Uh oh!

DarkSharpness commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DarkSharpness commented Jul 6, 2026 •

edited by github-actions Bot

Loading

DarkSharpness commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading