[JustinTong0323]

Per Lianmin's feedback (2026-06-19): make the default arguments just work — drop hand-set serve flags where the runtime default is correct or better, instead of overriding in the cookbook.

GLM-5.2 (zai-org/glm-5.2.jsx)

• --mem-fraction-static: keep it explicit (0.85 / 0.8 / 0.9). Earlier in this PR the flags were dropped to take the runtime auto-mfs, but verification showed the explicit values are the safer choice across the H200/B200/B300/GB300 cells, so they were put back.
• Drop --cuda-graph-max-bs (32/128/256 caps) — use the runtime default (512).
• --max-running-requests: drop on low-latency (spec defaults to 48) and high-throughput (let KV capacity bound it); keep a high 256 ceiling on balanced so KV capacity, not an artificial cap, bounds concurrency.
• Keep --chunked-prefill-size 32768 on balanced (the dominant prefill lever).
• GB300 high-throughput keeps env SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512. DeepEP's low_latency dispatch asserts x.size(0) <= num_max_dispatch_tokens_per_rank (deep_ep.cpp:1262, default 128). The decode cuda-graph capture builds a dummy batch of bs tokens per rank (not DP-split), so the 256/512 capture buckets trip the assert at default 128; GB300 (DP4) also hits bs/4 = 256 > 128 at c1024 runtime. Raising the buffer to 512 fixes both and lets HT drop --max-running-requests. The other HT cells (H200/B200/B300, DP8) split the batch 8 ways and the capture clamp is handled by --max-running-requests 256, so they need no env.

MiniMax-M3 (MiniMax-M3.mdx §2.1)

Corrected the note: --page-size 128 must stay explicit. The fa4 backend auto-forces page_size to 128 only when set via --prefill-attention-backend fa4; the M3 recipe sets the combined --attention-backend fa4, whose auto-force path is not covered, so omitting --page-size 128 would fall back to page size 1.

Note on MiniMax-M3 flags

The M3 cells keep --mem-fraction-static and --chunked-prefill-size unchanged. Verified on 8×B200 that dropping --mem-fraction-static (runtime default 0.807) OOMs at the prefill piecewise cuda-graph dummy forward — these are a hard constraint of the piecewise prefill graph (needs large activation headroom), not a redundant override.

Benchmark methodology

All GLM-5.2 speed numbers are re-measured on main @ 09ca4fc with real weights, --random-range-ratio 1.0 (the bench_serving default 0.0 randomizes each request's isl/osl in [1, max], which makes runs incomparable), and --flush-cache per run. sglang_version records the commit (main @ 09ca4fc) rather than the release tag. Spec cells pin the EAGLE acceptance length via env so the numbers are stable and comparable across runs: low-latency (5-1-6) uses SGLANG_SIMULATE_ACC_LEN=3.5 (match-expected: 50% accept 3 / 50% accept 4), balanced (1-1-2) uses =2 (accept 2 of 2). HT has no spec. --load-format dummy was verified to diverge from real weights (dummy is ~39% faster on spec cells, ~19% on HT — dummy weights skew MoE routing), so all cells use real weights.

Verification (public GLM-5.2-FP8, main @ 09ca4fc, real weights)

• GB300 (4×GB300, TP4) — low-latency: c1 51 / c16 264 tok/s/GPU; balanced: c64 493 / c256 756; HT: c1024 671. HT serve verified: with env ...=512 all 52 decode buckets capture and health=200; without the env the deep_ep.cpp:1262 assert trips at the bs=512 capture bucket, so the env is required for capture (not just runtime) and the recipe does not depend on the in-flight capture-clamp PR.
• B200 (8×B200, TP8) — low-latency: c1 32 / c16 164; balanced: c64 342 / c256 558; HT: c1024 451. Weights read from the verda cluster-storage model cache (no download). HT keeps --max-running-requests 256 (clamps the decode capture list to ≤32 < the 128 DeepEP buffer); no env buffer bump.
• H200 / B300: benchmark pending re-measurement (serve recipes are in glm-5.2.jsx; cells render "pending" until measured).
• MiniMax-M3 8×B200: patched command (no mfs) → OOM; original command → serve ready (1M context).

---

CI States

Latest PR Test (Base): ✅ Run #28347992858
Latest PR Test (Extra): ✅ Run #28347992791

[gemini-code-assist]

Code Review

This pull request updates the documentation for MiniMax-M3 to clarify the necessity of explicitly setting --page-size 128 when using --attention-backend fa4. Additionally, it cleans up several configurations in glm-5.2.jsx by removing --mem-fraction-static and --cuda-graph-max-bs flags, and increases --max-running-requests to 256 to avoid artificial concurrency limits. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[JustinTong0323]

/rerun-failed-ci