⭐ Must Read: sgl project/sglang/pull/

qiushixiaoyu · 2026-06-23T08:14:13Z

Motivation

This PR adds SM90 DeepGEMM MegaMoE adaptation for DeepSeek-V4 FP8 serving. It enables the MegaMoE A2A path with the DeepGEMM MoE runner on SM90, including the pre-dispatch JIT kernel, FP8 expert weight preparation, and DeepSeek-V4 integration.

The goal is to improve long-context / large decode serving throughput for DeepSeek-V4-Flash/Pro-FP8 workloads while keeping the path guarded by environment variables.

Modifications

Add SM90 MegaMoE pre-dispatch JIT kernel for preparing activations, scales, top-k indices, and top-k weights.
Add unit tests for the SM90 MegaMoE pre-dispatch kernel against a PyTorch reference implementation.
Integrate the MegaMoE path into DeepSeek-V4 / FP8 MoE execution.
Add FP8 MegaMoE expert weight preparation support.
Wire DeepGEMM MegaMoE execution through the existing MoE backend selection.

Environment variables used to enable the path:

SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 enables the DeepGEMM MegaMoE path.
SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 enables the fixed MegaMoE memory path.
SGLANG_OPT_USE_JIT_EP_ACTIVATION=1 is required by the fixed MegaMoE memory path.
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 configures the max token buffer size per rank used by the MegaMoE path. In PD colocated serving, it should be set to the same value as --chunked-prefill-size; in decode-only serving, it should be larger than the maximum number of tokens per rank. The experiment below used 4096, matching --chunked-prefill-size=4096.

Serving command used in the experiment:

export MODEL_PATH=/data00/models/DeepSeek-V4-Flash-FP8
export SERVER_PORT=30000
export NCCL_IB_GID_INDEX=3
export NCCL_IB_HCA=mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_8
export FLASHINFER_DISABLE_VERSION_CHECK=1
export NVSHMEM_HCA_PE_MAPPING=mlx5_1:1:1,mlx5_2:1:1,mlx5_3:1:1,mlx5_4:1:1,mlx5_5:1:1,mlx5_6:1:1,mlx5_7:1:1,mlx5_8:1:1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_THINKING=1
export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=0
export GLOO_SOCKET_IFNAME=eth0
export NCCL_MIN_NCHANNELS=24
export NCCL_IB_QPS_PER_CONNECTION=8

export SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1
export SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1
export SGLANG_OPT_USE_JIT_EP_ACTIVATION=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096

python -m sglang.launch_server
--trust-remote-code
--model-path "${MODEL_PATH}"
--tp 8
--ep-size 8
--chunked-prefill-size 4096
--moe-a2a-backend deepep
--moe-runner-backend deep_gemm
--deepep-mode auto
--cuda-graph-max-bs 32
--max-running-requests 32
--speculative-algo EAGLE
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--enable-metrics
--host 0.0.0.0
--port "${SERVER_PORT}"
--mem-fraction-static 0.75
--tool-call-parser deepseekv4
--reasoning-parser deepseek-v4

Accuracy Tests

sgl-eval run gpqa
--n-repeats 16 --max-tokens 200000
--temperature 1.0 --top-p 1.0 --thinking
--out-dir /sgl-workspace/logs
--base-url http://localhost:30000/v1

== gpqa ==
198 examples x 16 repeats | 13017.8s | 2620 tok/s | 34.1M tokens

pass@1[avg-of-16] = 88.64% +/- 1.28% (SEM 0.32%)
pass@16 = 96.46%
majority@16 = 90.15%
no_answer = 0.03%
stop_rate = 100.00%
truncated_rate = 0.00%
error_rate = 0.00%

Benchmark	Result
GSM8K	Accuracy 0.956, Invalid 0.000
GPQA	Score 0.90625, num_examples=32

Unit test:

python -m pytest -q python/sglang/jit_kernel/tests/test_mega_moe_pre_dispatch_sm90.py

Result:

10 passed, 20 warnings in 7.32s

Speed Tests and Profiling

DeepGEMM dependency note:

The FP8 MegaMoE operator has already been merged into DeepGEMM in sgl-project/DeepGEMM#36. This benchmark additionally includes the small-batch performance optimization from sgl-project/DeepGEMM#48.

Benchmark script:
The serving benchmark was run with sglang_auto_bench.py, an internal helper script that wraps python -m sglang.bench_serving and automatically searches for the highest request rate / max concurrency for a given input-output length. It reports both:

SLO-compliant throughput, constrained by the configured TTFT / TPOT targets.
Max throughput, which may violate the SLO and is used only to understand the saturation point.

For the result below, the script was run with random prompts, input/output length 3500/1500, and SLO targets TTFT <= 2000 ms and TPOT <= 20 ms.

Mode	SLO total throughput	SLO output throughput	TTFT	TPOT
MegaMoE ON	7001.55 tok/s	2100.46 tok/s	683.94 ms	13.83 ms
MegaMoE OFF	5727.56 tok/s	1718.27 tok/s	825.53 ms	17.05 ms

MegaMoE ON vs OFF:
SLO total throughput: +22.24%
SLO output throughput: +22.24%

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #30411502411
Latest PR Test (Extra): ✅ Run #30411502209

gemini-code-assist

Code Review

This pull request introduces the SM90 (Hopper) variant of the mega_moe_pre_dispatch kernel for DeepSeek v4, enabling all-FP8 Mega MoE support on SM90 GPUs. It includes the CUDA kernel implementation, JIT loading, integration into the mega_moe layer, and comprehensive unit tests. Feedback on the changes suggests optimizing _interleave_l1_weight_only by removing a redundant memory allocation and copy operation when stacking and reshaping tensors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

yuan-luo · 2026-06-24T06:04:20Z

Could we call the DeepGEMM MegaMoE API directly instead of porting all the MegaMoE kernel code piece?
sgl-project/DeepGEMM#36

Something like this:

    if _is_sm90():
        # SM90 (Hopper) fp8 mega MoE (DeepGEMM dev). Input is quantized on the
        # host and copied into the symm buffer (no mega_moe_pre_dispatch kernel,
        # which is the SM100 path).
        from deep_gemm.utils import per_token_cast_to_fp8

        if num_tokens > 0:
            x_fp8 = per_token_cast_to_fp8(
                hidden_states, use_ue8m0=False, gran_k=128, use_packed_ue8m0=False
            )
            buf.x[:num_tokens].copy_(x_fp8[0])
            buf.x_sf[:num_tokens].copy_(x_fp8[1])
            buf.topk_idx[:num_tokens].copy_(topk_ids_in)
            buf.topk_weights[:num_tokens].copy_(topk_weights_in)

        num_local_experts = num_experts // ep_group.size()
        cum_stats = torch.zeros(
            (num_local_experts,), dtype=torch.int32, device=hidden_states.device
        )
        y = torch.empty(
            (max(num_tokens, 1), hidden_size),
            dtype=torch.bfloat16,
            device=hidden_states.device,
        )
        swiglu_limit = getattr(moe.config, "swiglu_limit", None)
        deep_gemm.fp8_mega_moe(
            y,
            moe.experts.mega_l1_weights,
            moe.experts.mega_l2_weights,
            buf,
            cumulative_local_expert_recv_stats=cum_stats,
            recipe=(128, 128, 128),
            activation="swiglu",
            activation_clamp=swiglu_limit,
            fast_math=True,
        )
        y = y[:num_tokens]
        if not moe.experts.should_fuse_routed_scaling_factor_in_topk:
            y.mul_(moe.routed_scaling_factor)
        return y

yuan-luo · 2026-06-24T06:52:25Z

In build_mega_moe_experts_weights adding SM90 deep gemm weights processing.

def build_mega_moe_experts_weights(experts) -> None:
    if getattr(experts, "_mega_moe_weights_built", False):
        return
    if _is_sm90():
        # SM90 (Hopper) mega weight prep: gate/up interleave only (no UTCCP /
        # ue8m0 transpose, which are SM100-only). sglang stores fp8 weights in
        # an int8 container; view back to fp8_e4m3fn. Scales stay in the
        # loaded 128x128 fp8 block layout.
        import deep_gemm

        w13 = experts.w13_weight.data
        w2 = experts.w2_weight.data
        if w13.dtype == torch.int8:
            w13 = w13.view(torch.float8_e4m3fn)
        if w2.dtype == torch.int8:
            w2 = w2.view(torch.float8_e4m3fn)
        l1_pair, l2_pair = deep_gemm.transform_weights_for_mega_moe_sm90(
            (w13, experts.w13_weight_scale_inv.data),
            (w2, experts.w2_weight_scale_inv.data),
        )
        experts.mega_l1_weights = l1_pair
        experts.mega_l2_weights = l2_pair
        experts._mega_moe_weights_built = True
        return
    from deep_gemm import (
        transform_sf_into_required_layout,
        transform_weights_for_mega_moe,
    )
    from deep_gemm.mega import _interleave_l1_weights, _transpose_sf_for_utccp

Fridge003 · 2026-06-25T03:16:45Z

@qiushixiaoyu Please test the full set of GPQA with 32 turns, thanks
Instructions:

sgl-eval run gpqa \
  --model fp8_flash --api-key <api-key> \
  --n-repeats 16 --max-tokens 200000 \
  --temperature 1.0 --top-p 1.0 --thinking \
  --out-dir /sgl-workspace/logs \
  --base-url http://localhost:30000/v1

The model should be launched with SGLANG_DEFAULT_THINKING=1

Fridge003 · 2026-06-25T03:31:00Z

Is sm90 fp8 megamoe also applicable on other models, like dsv3 or glm5?

qiushixiaoyu · 2026-06-25T07:06:20Z

Is sm90 fp8 megamoe also applicable on other models, like dsv3 or glm5?

I think it's feasible, but we'll need to make some code adaptations.

qiushixiaoyu · 2026-06-26T06:59:32Z

@qiushixiaoyu Please test the full set of GPQA with 32 turns, thanks Instructions:
sgl-eval run gpqa \
  --model fp8_flash --api-key <api-key> \
  --n-repeats 16 --max-tokens 200000 \
  --temperature 1.0 --top-p 1.0 --thinking \
  --out-dir /sgl-workspace/logs \
  --base-url http://localhost:30000/v1
The model should be launched with SGLANG_DEFAULT_THINKING=1

sgl-eval run gpqa
--n-repeats 16 --max-tokens 200000
--temperature 1.0 --top-p 1.0 --thinking
--out-dir /sgl-workspace/logs
--base-url http://localhost:30000/v1

== gpqa ==
198 examples x 16 repeats | 13017.8s | 2620 tok/s | 34.1M tokens

pass@1[avg-of-16] = 88.64% +/- 1.28% (SEM 0.32%)
pass@16 = 96.46%
majority@16 = 90.15%
no_answer = 0.03%
stop_rate = 100.00%
truncated_rate = 0.00%
error_rate = 0.00%

github-actions · 2026-07-02T07:43:08Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py

🚀 8-gpu-h200 (2 tests): ❌ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py
cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py

qiushixiaoyu · 2026-07-03T05:00:45Z

@Fridge003
I've root-caused the h200 failure.
The 8-gpu-h200 TestDSV4FlashFP8H200MegaMoE failure is a missing DeepGEMM dependency, not a bug in this PR's logic. The CI image's deep_gemm predates sgl-project/DeepGEMM#48, so the SM90 FP8 Mega-MoE path is never actually activated; execution silently falls back into the generic masked GEMM, which aborts on the first idle (empty) batch.
The PR description already lists #48 as a small-batch perf optimization — but the h200 failure shows it's actually a hard runtime dependency for the SM90 path: #48 also adds the mega_moe_pre_dispatch_sm90 entrypoint that is_sm90_fp8_mega_moe_available() gates on.

laixinn · 2026-07-07T07:04:58Z

@qiushixiaoyu Hi, I checkout to your branch, install sgl-deep-gemm==0.1.4, sglang-kernel==0.4.4, and flashinfer_python==0.6.8.post1, with cu129 version. Running the provided script throws an error during cuda graph capture:

[2026-07-07 14:59:27 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 4261, in run_scheduler_process
    scheduler = Scheduler(
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 420, in __init__
    self.init_model_worker()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 863, in init_model_worker
    self.init_all_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 846, in init_all_cuda_graphs
    self.tp_worker.init_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/tp_worker.py", line 351, in init_cuda_graphs
    self.model_runner.init_cuda_graphs(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 926, in init_cuda_graphs
    self.init_decode_cuda_graph()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 2608, in init_decode_cuda_graph
    self.decode_cuda_graph_runner = graph_runners[self.device](self)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 355, in __init__
    self.capture()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 692, in capture
    self._capture_one_stream()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 743, in _capture_one_stream
    self.capture_one_shape(bs, forward, stream_idx, variant_label)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 838, in capture_one_shape
    self.backend.capture_one(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner_backend/full_cuda_graph_backend.py", line 92, in capture_one
    forward_fn()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 807, in run_once
    out = forward(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 2035, in forward
    hidden_states = self.model.forward(
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1874, in forward
    hidden_states, prev_residual, prev_post, prev_comb = layer(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1711, in forward
    attn_tp_all_gather(gathered, hidden_states.contiguous())
  File "/workdir/sglang-dsv4/python/sglang/srt/layers/dp_attention.py", line 740, in attn_tp_all_gather
    return get_attention_tp_group().all_gather(input, output_tensor_list=output_list)
  File "/workdir/sglang-dsv4/python/sglang/srt/distributed/parallel_state.py", line 1046, in all_gather
    return torch.distributed.all_gather(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4079, in all_gather
    work = group.allgather([tensor_list], [tensor], opts)
ValueError: Tensor input and output of _broadcast_oop must have the same number of elements

qiushixiaoyu · 2026-07-08T03:42:18Z

@qiushixiaoyu Hi, I checkout to your branch, install sgl-deep-gemm==0.1.4, sglang-kernel==0.4.4, and flashinfer_python==0.6.8.post1, with cu129 version. Running the provided script throws an error during cuda graph capture:

[2026-07-07 14:59:27 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 4261, in run_scheduler_process
    scheduler = Scheduler(
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 420, in __init__
    self.init_model_worker()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 863, in init_model_worker
    self.init_all_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 846, in init_all_cuda_graphs
    self.tp_worker.init_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/tp_worker.py", line 351, in init_cuda_graphs
    self.model_runner.init_cuda_graphs(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 926, in init_cuda_graphs
    self.init_decode_cuda_graph()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 2608, in init_decode_cuda_graph
    self.decode_cuda_graph_runner = graph_runners[self.device](self)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 355, in __init__
    self.capture()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 692, in capture
    self._capture_one_stream()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 743, in _capture_one_stream
    self.capture_one_shape(bs, forward, stream_idx, variant_label)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 838, in capture_one_shape
    self.backend.capture_one(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner_backend/full_cuda_graph_backend.py", line 92, in capture_one
    forward_fn()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 807, in run_once
    out = forward(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 2035, in forward
    hidden_states = self.model.forward(
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1874, in forward
    hidden_states, prev_residual, prev_post, prev_comb = layer(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1711, in forward
    attn_tp_all_gather(gathered, hidden_states.contiguous())
  File "/workdir/sglang-dsv4/python/sglang/srt/layers/dp_attention.py", line 740, in attn_tp_all_gather
    return get_attention_tp_group().all_gather(input, output_tensor_list=output_list)
  File "/workdir/sglang-dsv4/python/sglang/srt/distributed/parallel_state.py", line 1046, in all_gather
    return torch.distributed.all_gather(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4079, in all_gather
    work = group.allgather([tensor_list], [tensor], opts)
ValueError: Tensor input and output of _broadcast_oop must have the same number of elements

@laixinn , Could you share the full launch command and all related environment variables?
Especially: --tp/--dp/--enable-dp-attention, --cuda-graph-, --moe-a2a-backend, --moe-runner-backend, speculative decoding args, and SGLANG_OPT_ / SGLANG_DSV4_* envs.

Also please confirm whether these return True:
python - <<'PY'
import deep_gemm
print("fp8_mega_moe", hasattr(deep_gemm, "fp8_mega_moe"))
print("mega_moe_pre_dispatch_sm90", hasattr(deep_gemm, "mega_moe_pre_dispatch_sm90"))
PY

laixinn · 2026-07-10T03:45:02Z

@qiushixiaoyu Hi, I checkout to your branch, install sgl-deep-gemm==0.1.4, sglang-kernel==0.4.4, and flashinfer_python==0.6.8.post1, with cu129 version. Running the provided script throws an error during cuda graph capture:

[2026-07-07 14:59:27 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 4261, in run_scheduler_process
    scheduler = Scheduler(
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 420, in __init__
    self.init_model_worker()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 863, in init_model_worker
    self.init_all_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/scheduler.py", line 846, in init_all_cuda_graphs
    self.tp_worker.init_cuda_graphs()
  File "/workdir/sglang-dsv4/python/sglang/srt/managers/tp_worker.py", line 351, in init_cuda_graphs
    self.model_runner.init_cuda_graphs(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 926, in init_cuda_graphs
    self.init_decode_cuda_graph()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/model_runner.py", line 2608, in init_decode_cuda_graph
    self.decode_cuda_graph_runner = graph_runners[self.device](self)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 355, in __init__
    self.capture()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 692, in capture
    self._capture_one_stream()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 743, in _capture_one_stream
    self.capture_one_shape(bs, forward, stream_idx, variant_label)
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 838, in capture_one_shape
    self.backend.capture_one(
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner_backend/full_cuda_graph_backend.py", line 92, in capture_one
    forward_fn()
  File "/workdir/sglang-dsv4/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 807, in run_once
    out = forward(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 2035, in forward
    hidden_states = self.model.forward(
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1874, in forward
    hidden_states, prev_residual, prev_post, prev_comb = layer(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workdir/sglang-dsv4/python/sglang/srt/models/deepseek_v4.py", line 1711, in forward
    attn_tp_all_gather(gathered, hidden_states.contiguous())
  File "/workdir/sglang-dsv4/python/sglang/srt/layers/dp_attention.py", line 740, in attn_tp_all_gather
    return get_attention_tp_group().all_gather(input, output_tensor_list=output_list)
  File "/workdir/sglang-dsv4/python/sglang/srt/distributed/parallel_state.py", line 1046, in all_gather
    return torch.distributed.all_gather(
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/workdir/sglang-dsv4/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4079, in all_gather
    work = group.allgather([tensor_list], [tensor], opts)
ValueError: Tensor input and output of _broadcast_oop must have the same number of elements

@laixinn , Could you share the full launch command and all related environment variables? Especially: --tp/--dp/--enable-dp-attention, --cuda-graph-, --moe-a2a-backend, --moe-runner-backend, speculative decoding args, and SGLANG_OPT__ / SGLANG_DSV4* envs.

Also please confirm whether these return True: python - <<'PY' import deep_gemm print("fp8_mega_moe", hasattr(deep_gemm, "fp8_mega_moe")) print("mega_moe_pre_dispatch_sm90", hasattr(deep_gemm, "mega_moe_pre_dispatch_sm90")) PY

I got

fp8_mega_moe True
mega_moe_pre_dispatch_sm90 False

Which sgl-deep-gemm version should I install? Or I should build the sgl-project/DeepGEMM dev branch?

qiushixiaoyu · 2026-07-13T03:37:10Z

@laixinn Please use the dev branch of sgl-project/DeepGEMM. I found that the changes in sglang PR https://github.com/sgl-project/sglang/pull/25835/changes cause an accuracy drop when MegaMOE is enabled.

usctjm · 2026-07-20T09:41:19Z

@qiushixiaoyu Hi，Have you verified the correctness with a max_token_per_rank=8K? I observed a significant drop in AIME results when running AIME25 with a concurrency of 40，max_token_per_rank=8K（sgl-version: 0.5.15，tp8 + ep8 + cp，run deepseek v4 flash）Or am I using it incorrectly somewhere?

Fridge003 · 2026-07-25T00:39:36Z

@qiushixiaoyu DeepGemm should be updated to include all the changes
Can you please rebase

qiushixiaoyu · 2026-07-27T03:40:57Z

@Fridge003 OK, I'll rebase and re-run the accuracy tests.

qiushixiaoyu · 2026-07-29T02:33:19Z

/rerun-failed-ci

hqliuuu · 2026-07-29T02:40:47Z

Hi, after the latest rebase and DeepGEMM update, has the MegaMoE accuracy-drop issue related to SGLang PR #25835 been fixed or re-verified? Could you share the exact SGLang/DeepGEMM commits and evaluation configuration, especially fused top-k on/off, TP/EP/CP, max tokens per rank, concurrency, and dataset? Thanks!

qiushixiaoyu · 2026-07-29T03:36:27Z

@hqliuuu
Thanks for the questions.

Accuracy drop: it has been fixed and the fix is already merged into the DeepGEMM
repo, so the latest sglang image is sufficient — no extra patching needed. Our runs
used the DeepGEMM shipped in lmsysorg/sglang:dev.

Commits

SGLang: f75fd7db2b (this PR rebased onto upstream main 32c30c0f96)
DeepGEMM: as shipped in the lmsysorg/sglang:dev image

Config: 8×H20 (SM90) · TP8 / EP8 / DP-attention 8 · CP not enabled ·
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=1280 ·
--max-running-requests 32 · fused top-k on
(default SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=True, not overridden).

Launch:

export MODEL_PATH=/data00/models/DeepSeek-V4-Flash-FP8
export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=0
export GLOO_SOCKET_IFNAME=eth0
export NCCL_IB_GID_INDEX=3
export NCCL_IB_HCA=mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_8
export FLASHINFER_DISABLE_VERSION_CHECK=1

export SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1
export SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1
export SGLANG_OPT_USE_JIT_EP_ACTIVATION=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=1280
export SGLANG_DEFAULT_THINKING=1

python -m sglang.launch_server \
  --trust-remote-code \
  --model-path "${MODEL_PATH}" \
  --tp-size 8 --ep-size 8 \
  --enable-dp-attention --dp-size 8 --enable-dp-lm-head \
  --chunked-prefill-size 8192 \
  --moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
  --deepep-mode auto \
  --cuda-graph-max-bs 32 --max-running-requests 32 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.75 \
  --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 --port 30000

qiushixiaoyu · 2026-07-29T05:06:09Z

/rerun-failed-ci

qiushixiaoyu · 2026-07-30T02:41:46Z

/rerun-failed-ci

Co-authored-by: yinding <yinding@bytedance.com>

qiushixiaoyu requested review from AniZpZ, BBuf, DarkSharpness, Edwardf0t1, FlamingoPg, Fridge003, HaiShaw, HydraQYH, Ying1123, b8zhong, celve, ch-wan, ispobock, merrymercy and yuan-luo as code owners June 23, 2026 08:14

github-actions Bot added deepseek jit-kernel labels Jun 23, 2026

gemini-code-assist Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread python/sglang/srt/layers/moe/mega_moe.py Outdated

qiushixiaoyu force-pushed the mega_moe_adapt branch from 29360d0 to 2d5188b Compare June 23, 2026 08:30

Fridge003 added the release-highlight Candidate PR for release note highlight label Jun 23, 2026

Fridge003 self-assigned this Jun 23, 2026

Fridge003 requested changes Jun 25, 2026

View reviewed changes

qiushixiaoyu force-pushed the mega_moe_adapt branch from b3c6135 to f1ba22e Compare June 25, 2026 11:04

Fridge003 approved these changes Jun 29, 2026

View reviewed changes

amd-bot mentioned this pull request Jul 5, 2026

[CI Monitor] Daily Report - 2026-07-05 bingxche/sglang-ci-bot#126

Open

qiushixiaoyu force-pushed the mega_moe_adapt branch from c88a9d9 to b6c3549 Compare July 18, 2026 09:58

yinding added 2 commits July 28, 2026 21:17

Add SM90 FP8 MegaMoE adaptation

ab69621

Add SM90 FP8 MegaMoE cookbook configuration tips and H200 e2e subtest

f75fd7d

qiushixiaoyu force-pushed the mega_moe_adapt branch from b6c3549 to f75fd7d Compare July 28, 2026 13:21

qiushixiaoyu requested review from Alisehen and OrangeRedeng as code owners July 28, 2026 13:21

Fridge003 added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Jul 29, 2026

Merge branch 'main' into mega_moe_adapt

2cef726

Fridge003 merged commit fc007e1 into sgl-project:main Jul 30, 2026
265 of 340 checks passed

saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026

Add SM90 FP8 MegaMoE support for DeepSeek-V4 (sgl-project#29016)

9a2d96e

Co-authored-by: yinding <yinding@bytedance.com>

hnyls2002 mentioned this pull request Aug 18, 2026

[feat] Enable DeepGEMM Mega MoE as MoE layer runner backend in NVLink domain #23167

Closed

5 tasks

lengrongfu mentioned this pull request Aug 24, 2026

[DSV4][SM90] add dsv4 fp8 weight use mega_moe backend vllm-project/vllm#53527

Open

4 tasks

Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026

Add SM90 FP8 MegaMoE support for DeepSeek-V4 (sgl-project#29016)

16a70b9

Co-authored-by: yinding <yinding@bytedance.com>

Conversation

qiushixiaoyu commented Jun 23, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

yuan-luo commented Jun 24, 2026

Uh oh!

yuan-luo commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fridge003 commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fridge003 commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiushixiaoyu commented Jun 25, 2026

Uh oh!

qiushixiaoyu commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiushixiaoyu commented Jul 3, 2026

Uh oh!

laixinn commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiushixiaoyu commented Jul 8, 2026

Uh oh!

laixinn commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiushixiaoyu commented Jul 13, 2026

Uh oh!

usctjm commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fridge003 commented Jul 25, 2026

Uh oh!

qiushixiaoyu commented Jul 27, 2026

Uh oh!

qiushixiaoyu commented Jul 29, 2026

Uh oh!

hqliuuu commented Jul 29, 2026

Uh oh!

qiushixiaoyu commented Jul 29, 2026

Uh oh!

qiushixiaoyu commented Jul 29, 2026

Uh oh!

qiushixiaoyu commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qiushixiaoyu commented Jun 23, 2026 •

edited by github-actions Bot

Loading

yuan-luo commented Jun 24, 2026 •

edited

Loading

Fridge003 commented Jun 25, 2026 •

edited

Loading

github-actions Bot commented Jul 2, 2026 •

edited

Loading

laixinn commented Jul 7, 2026 •

edited

Loading

laixinn commented Jul 10, 2026 •

edited

Loading

usctjm commented Jul 20, 2026 •

edited

Loading