[BingooYang]

Motivation

OpenAIServingBase.handle_request performs request conversion inline on the serving event loop. That conversion is not cheap: for /v1/chat/completions it runs the message processor and the chat template (jinja) and produces the full tokenized prompt. For a 100k+ token prompt this occupies the event loop for hundreds of milliseconds.

While the loop is blocked, everything else the server is doing stalls with it:

• new connections are not accepted,
• already-open streaming responses are not flushed, so unrelated in-flight requests see their inter-token latency spike,
• health/liveness endpoints sharing the loop stop responding.

The component that does the heavy part is the Rust tokenizer, which releases the GIL while encoding, so this work can genuinely overlap with the event loop instead of serializing behind it. The jinja render is still Python and still competes for the GIL, but the loop keeps running instead of waiting for the whole conversion to finish.

Modifications

Single file, python/sglang/srt/entrypoints/openai/serving_base.py:

• In handle_request, _convert_to_internal_request is now awaited through loop.run_in_executor(None, ...) instead of being called inline.

The conversion logic itself is untouched, and the returned adapted_request /processed_request are the same objects as before — only the thread they are produced on changes. The surrounding flow is unchanged: validation still runs on the loop, received_time is still stamped before dispatch, and the streaming / non-streaming branches below are unaffected. The default executor is used, matching the existing pattern in http_request_decompression.py.

Accuracy Tests

Not applicable. This PR does not touch the model forward path, kernels, sampling, or the tokenizer itself. The same conversion runs on the same inputs, so model outputs are unchanged; the only difference is which thread executes it.

Speed Tests and Profiling

Deployment: DeepSeek-V4-Flash, PD-disaggregated dwdp serving config,
--max-running-requests 480.

Workload: long-context, average input length 265k tokens.

• QPS: +3% with this patch (same node, same server args).

The gain comes from the event loop no longer serializing on prompt conversion: with 265k-token inputs, jinja rendering plus full-prompt tokenization holds the loop for long enough that accepting new requests and flushing in-flight streams are both delayed, so the server is not saturated on the compute side even at max-running-requests 480.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #38264170635
Latest PR Test (Extra): ⏳ Run #38264170516
Latest PR Test (AMD ROCm 10): ❌ Run #38264170600

[BingooYang]

/rerun-failed-ci

[JustinTong0323]

Deleting unnecessary test.

[JustinTong0323]

LGTM

[JustinTong0323]

/rerun-failed-ci