[mickqian]

Stacked on #43361 (#43164 is merged). Merge #43361 first; this PR's own change is the cuDNN gate in minimax_h3.py and one cookbook line.

Motivation

On Blackwell, MiniMax-H3's dense attention runs on cuDNN SDPA through DynamicCudnnSDPAImpl. The pipelined Ulysses exchange from #43164 only accepted the FlashAttention backend, so B200/B300 kept the sequential exchange and got none of it.

Modifications

• The pipeline also accepts AttentionBackendEnum.DYNAMIC_CUDNN_SDPA. Attention still runs through the backend's forward_varlen, one head group at a time.
• The skip-softmax check only exists on the FlashAttention impl, so it is now optional.
• Cookbook: the configuration note says the pipeline runs against FlashAttention on Hopper and cuDNN SDPA on Blackwell.

Accuracy Tests

• Per head group, cuDNN SDPA gives the same bytes as those heads inside the full call. Checked with torch.equal on 4×B300 at H3's per-rank attention shapes:

  • 49652 tokens × 28 heads in groups of 4 (Ref2VA, Ulysses 2);
  • 37742 tokens × 28 heads in groups of 4 (T2VA, Ulysses 2);
  • 37744 tokens × 14 heads in groups of 2 (T2VA, Ulysses 4).

  It holds on both torch 2.14.1 with cuDNN 9.24 and torch 2.13 with cuDNN 9.20.

• End to end on 4×B300, the MP4 md5 is the same for the sequential exchange and for 7, 4 and 2 groups, in every cell.

Speed Tests and Profiling

Client end-to-end latency, 4×B300, 50 steps, default quality, one run per arm. The base is this stack (#43164 + #43361) with #43327 and #43337, and SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS selects the sequential exchange (0) or a group count:

Case	sequential	7 groups (default)	4 groups	2 groups
T2VA 5 s, Ulysses 2	65.83 s	63.24 s (−3.9%)	63.40 s (−3.7%)	63.48 s (−3.6%)
T2VA 5 s, Ulysses 4	34.20 s	32.75 s (−4.3%)	—	32.90 s (−3.8%)
Ref2VA 5 s, Ulysses 2	102.68 s	99.24 s (−3.4%)	98.70 s (−3.9%)	98.89 s (−3.7%)

• Blackwell gains more than Hopper (2–3%) because its exchange is a larger share of a faster step.
• Splitting heads costs cuDNN some efficiency, which FlashAttention 3 does not pay. On B300, 7 group calls over the Ref2VA shape take 3.5% longer than one call over all heads, and over the Ulysses 4 shape (2 heads per call) 5.4% longer. That is why 4 groups edge out 7 for Ref2VA in this single-run pass.
• A paired ABBA of 7 against 4 groups on B300 puts 4 ahead by only 0.1–0.2%, within run-to-run noise: Ref2VA 98.70 → 98.51 s, T2VA 63.74 → 63.68 s. So the automatic count stays at 7 for both backends.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests. (The pipeline's parity test is backend-agnostic; the cuDNN equality was checked on B300 as above.)
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #38011817567
Latest PR Test (Extra): ❌ Run #38011817456
Latest PR Test (AMD ROCm 10): ⏳ Run #38011817643

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
sglang-doc	🟢 Ready	View Preview	Oct 10, 2026, 1:06 AM

💡 Tip: Enable Automations to automatically generate PRs for you.