[weireweire]

Motivation

--tokenizer-backend fastokens cannot serve DeepSeek-V4.1 with the fastokens version SGLang currently allows, and the official images do not ship fastokens at all.

The current pin breaks DeepSeek-V4.1. The fastokens extra is pinned to >=0.1.1,<0.2.0, so installs resolve to 0.1.2. That release has two bugs:

• It counts vocab_size as base vocab plus added tokens. Added tokens whose ids sit inside the base vocab are counted twice.
• decode raises on unknown ids. HF tokenizers returns "".

The DeepSeek-V4.1 tokenizer has 128000 BPE tokens and 1283 added tokens. Three of the added tokens use ids 0-2. So with the fastokens backend, len(tokenizer) is 129283 instead of 129280. At model init, the engram token map (build_compressed_token_map) decodes every id in range(len(tokenizer)). It fails on id 129280:

ValueError: decode error: unknown token ID: 129280

Both bugs are fixed upstream in fastokens 0.3.2 (crusoecloud/fastokens#55, crusoecloud/fastokens#65).

fastokens is not in the image. The extra is only pulled in by test, not by all. The Docker images install .[all], so a deployment has to pip install fastokens at startup before it can use the backend. The Rust frontends already link fastokens through dynamo-tokenizers.

Why it matters. The chat endpoint re-encodes the whole conversation on every turn. For agentic traces that is 300k-600k tokens per request. HF tokenizers encodes on one thread at about 1.8 us/token on Grace, so encoding takes about 0.6 s. That time is on the TTFT path and it blocks the API server event loop.

Modifications

• python/pyproject.toml: require fastokens>=0.3.2 (lower bound only, like most dependencies here).
• python/pyproject.toml: add sglang[fastokens] to the all extra, so images built from .[all] include it. The default --tokenizer-backend stays huggingface.

Accuracy Tests

All checks use the DeepSeek-V4.1-Flash tokenizer with fastokens 0.3.2 and 0.3.3, patched into transformers the same way --tokenizer-backend fastokens does it.

• len(tokenizer) is 129280, the same as HF. Decoding id 129280 returns "".
• Decoding each id in range(len(tokenizer)) one at a time, as the engram map does, gives the same text as HF for every id.
• Encoding rendered DeepSeek-V4.1 agentic chat prompts of 55k, 340k and 557k tokens gives ids identical to HF.
• End to end: GB300 TP4 DeepSeek-V4.1-Flash starts with --tokenizer-backend fastokens and fastokens 0.3.2. Model init, including the engram map, completes, and AgentX warmup and measured requests run without errors. With 0.1.2 the same launch fails at init with the error above.
• The existing test_hf_transformers_fastokens.py uses fastokens._compat._TokenizerShim and patch_transformers. Both are still present in 0.3.2 and 0.3.3.

Speed Tests and Profiling

Encode time (DeepSeek-V4.1 tokenizer, rendered agentic chat prompts, single call, CPU):

prompt tokens	HF tokenizers	fastokens 0.3.2
55k	0.044 s	0.004 s
340k	0.320 s	0.017 s
557k	0.498 s	0.029 s

Serving. GB300 TP4, DeepSeek-V4.1-Flash, DSpark speculative decoding, AgentX coding traces at concurrency 1, 30-minute measured window. Same image and flags; the only difference is --tokenizer-backend (fastokens 0.3.2 installed).

	huggingface	fastokens
TTFT p50	0.811 s	0.289 s
TTFT p90	1.490 s	1.398 s
completed requests	197	202
failed requests	0	0
TPOT p50	1.574 ms	1.573 ms
server prefix cache hit rate	0.961	0.960

Decode speed is unchanged. The remaining TTFT is prefill compute (about 0.23 s).

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): 🚫 Run #38200234622
Latest PR Test (Extra): ❌ Run #38200234581
Latest PR Test (AMD ROCm 10): ✅ Run #38200234720

[weireweire]

/tag-and-rerun-ci

[nvpohanh]

cc @Oasis-Git

[nvpohanh]

/rerun-failed-ci

[nvpohanh]

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

[mmangkad]

/rerun-test test/registered/unit/utils/test_hf_transformers_fastokens.py

[github-actions]

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/utils/test_hf_transformers_fastokens.py

[weireweire]

test pass, could we merge?

[YAMY1234]

This change is important for perf. TTFT, mean: 498 → 125 ms at concurrency 4, 501 → 112 ms at 8, 738 → 244 ms at 16. Total throughput +6.5% at concurrency 4. cc @Oasis-Git

DeepSeek-V4.1-Flash on 4×GB300 (TP4/EP4, DSpark), AgentX(393 traces, about 186K input tokens per turn on average). The only change between the two runs is --tokenizer-backend fastokens (fastokens 0.3.3); each run is a full 1-hour window.

[nvpohanh]

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

[nvpohanh]

[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/38193715150/job/114634289012) is not caused by this PR. It is a known breakage on main, tracked in #43746. Proposed fix: #43747. CI for this PR is blocked until that is fixed.