[SYChen123]

Motivation

When enabling BCG on deepseek v4, C4 indexer leads to heavy memory pressure. The root reason is that the indexer logits with shape of [N, context_length] is padded to [N, max_position_embeddings]. For deepseek v4, model's max_position_emb is 1048576. When N=8192, the logits itself will cost 8192 * 1048576 // 4 * 4 = 8GB.
Therefore, we propose to add a new parameter named --cuda-graph-prefill-max-context and the logit is padded to the user-specified max context size instead model's maximum context.

Modifications

1. Add --cuda-graph-prefill-max-context parameter which accepts only 1 value and currently only takes effect on deepseek v4. The c4_max_seq_len used in C4 indexer is padded to this user-specified value, rather than model's maximum context.
2. During replay, if the kv context length of any of the request exceeds the --cuda-graph-prefill-max-context, fallback to eager.

Accuracy Tests

gsm8k test on deepseek-v4-flash-0731 is good.

Speed Tests and Profiling

Memory usage reduction

Before this pr, capture will cost 43GB when --cuda-graph-max-bs-prefill=16384.

With this pr, the capture memory cost is reduced to 18GB. 58% memory reduction.

E2E test on deepseek-v4-flash-0731

E2E latency has 10% performance gains with BCG enabled. Without this pr, the server will OOM when chunk-prefill-size and max-captured-size are both set to 16384.

Setup	TTFT / TPOT (ms)	E2E (ms)	Input throughput (token/s)
Baseline	1075.57 / 1.68	1912.69	278674.46
With this pr	830.93 / 1.82	1737.34	306275.00

Server launching and benchmark commands are shown below.

# launch server
sglang serve \
    --trust-remote-code \
    --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
    --tp 2 \
    --moe-runner-backend flashinfer_mxfp4 \
    --speculative-algorithm DSPARK \
    --chunked-prefill-size 16384 \
    --max-prefill-tokens 16384 \
    --cuda-graph-backend-prefill breakable \
  	--cuda-graph-max-bs-prefill 16384 \
	--cuda-graph-prefill-max-context 262144 \
    --enable-deepseek-v4-fp4-indexer \
    --max-running-requests 256 \
    --swa-full-tokens-ratio 0.1 \
    --mem-fraction-static 0.75 \
    --reasoning-parser deepseek-v4 \
    --tool-call-parser deepseekv4 \
    --enable-strict-thinking \
    --enable-hierarchical-cache \
    --hicache-ratio 2 \
    --enable-cache-report \
    --enable-metrics \
    --log-requests \
    --log-requests-level 1 \
    --host 192.168.153.127 \
    --port 9090 > server-0.log 2>&1 &

# benchmark
python3 -m sglang.bench_serving \
            --model deepseek-ai/DeepSeek-V4-Flash-0731 \
            --flush-cache \
            --host "${HOST}" \
            --port "${PORT}" \
            --backend sglang \
            --dataset-name generated-shared-prefix \
            --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
            --num-prompts 512 \
            --gsp-num-groups 1 \
            --gsp-prompts-per-group 512 \
            --gsp-system-prompt-len 61440 \
            --gsp-question-len 4096 \
            --gsp-output-len 500 \
            --max-concurrency 32 \
            --warmup-requests 10 \
            > "benchmark.log" 2>&1 
    done
done

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #34568924841
Latest PR Test (Extra): ✅ Run #34568924736
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[Oasis-Git]

Let's run ci

[Oasis-Git]

/rerun-failed-ci

[Oasis-Git]

/rerun-failed-ci

[Oasis-Git]

/rerun-failed-ci

[Oasis-Git]

extra pass: https://github.com/sgl-project/sglang/actions/runs/34568924736

[Oasis-Git]

/rerun-failed-ci

[Oasis-Git]

base pass: https://github.com/sgl-project/sglang/actions/runs/34568924841