[HJSang]

This is the main counterpart of #42838 on sglang-miles, which becomes its cherry-pick once this merges. The diff is identical; the measurements and the end-to-end check below were done with it on sglang-miles (f19dcfb12), and the tests were rerun on main.

Motivation

RL clients that train on top-k candidates, such as score centering in miles, read meta_info.output_top_logprobs. That field is completion_tokens rows of k [logprob, token_id, text] triples. Producing and consuming it is expensive at every hop:

• Detokenization. The TokenizerManager decodes every candidate id to text, one batch_decode call per position.
• A second copy. With logprobs, the OpenAI chat response also builds the same candidates as choices[].logprobs.content[].top_logprobs pydantic objects, each carrying a token string and a bytes list.
• Size. At 160 tokens × k=128 the body is about 2.6 MB of JSON. The client then parses it back into Python objects.
• Non-finite values. JSON can't carry -inf, so a non-finite candidate logprob arrives as null.

Prompt top logprobs already have a compact format: return_flat_raw_top_logprobs (+ _b64). This PR adds the output-side counterpart.

Modifications

• GenerateReqInput.return_flat_raw_output_top_logprobs. On the finished response, meta_info carries output_top_logprobs_{val,idx}_flat (or _flat_b64 with _dtype), output_top_logprobs_shape and output_top_logprobs_null_prefix. The layout is identical to the prompt fields. Base64 packs float32/int32 arrays with the same helpers; plain JSON flattens the engine's own rows without a NumPy round trip, as the prompt side does (suggested by @ByronHsu). The nested rows are neither detokenized nor returned.
• Final response only. The arrays are built once, at finish. That includes the abort path, which passes finished=True. Intermediate non-streaming outputs carry no output top logprobs, and stream=True is rejected.
• Fallback. Rows that are not rectangular fall back to the nested format with an error log, the same as the prompt side, so a bad request can't break the shared batch-output loop.
• return_flat_raw_top_logprobs_b64 now applies to either flat flag.
• Chat completions. ChatCompletionRequest exposes return_flat_raw_output_top_logprobs, which requires return_meta_info=true, and return_flat_raw_top_logprobs_b64. On chat the b64 flag defaults to true, so return_flat_raw_output_top_logprobs alone returns base64; set it to false for plain JSON lists. /generate and the prompt-side flat format keep false as the default. With the flag, choices[].logprobs keeps the selected-token logprobs, but its top_logprobs lists are empty. The candidates are only in meta_info.
• Unchanged. The scheduler and the detokenizer still ship per-step rows as before; only TokenizerManager assembly changes.
• API spec. return_flat_raw_output_top_logprobs joins return_flat_raw_top_logprobs(_b64) in test_parity.py's Python-only request fields; the v1 HTTP contract hasn't adopted these flags.

Gateways that parse chat requests into a typed struct must forward the two new fields. For the miles gateway that is radixark/sgl-router-for-miles#24. Consumer: radixark/miles#3952.

Accuracy Tests

There is no change to the values: float32 logprobs and int32 ids are bit-exact, and -inf is preserved. I replayed one response across repositories on the H200 host:

• SGLang side: the real TokenizerManager.convert_logprob_style, fed in 50-token chunks (the non-streaming force-stream interval), then OpenAIServingChat._build_chat_response, then JSON.
• miles side: json.loads and its append_rollout_topk_logprobs decoder.

The decoded [tokens, k] arrays equal the engine rows for both formats. With a -inf candidate, only the flat format decodes back to -inf; the nested format arrives as null.

Speed Tests and Profiling

Same replay, CPU only, H200 host, Qwen3.5-9B tokenizer, median of 5:

tokens × k	format	TM detokenize	build response	FastAPI serialize (model_dump_json)	body	client parse + decode
160 × 128	nested	42.7 ms	70.4 ms	323.9 ms (53.2)	2.62 MB	40.3 ms
160 × 128	flat b64	1.4 ms	0.8 ms	5.0 ms (0.8)	0.24 MB	0.8 ms
1000 × 32	nested	70.0 ms	110.7 ms	906.3 ms (87.4)	4.22 MB	67.5 ms
1000 × 32	flat b64	3.9 ms	3.7 ms	20.0 ms (3.0)	0.50 MB	2.7 ms
1000 × 128	nested	261.5 ms	895.4 ms	2598.3 ms (383.9)	16.39 MB	297.6 ms
1000 × 128	flat b64	8.5 ms	4.7 ms	30.5 ms (4.6)	1.52 MB	4.7 ms

The FastAPI column is the current serialization path; the bracketed value is the single-pass serialization from #42889 (#42834 on sglang-miles). The two PRs are independent. All of this CPU work runs on the HTTP/TokenizerManager event loop.

End-to-end on H200

Measured on sglang-miles (f19dcfb12). Stack P is this change (#42838) + #42834 + the gateway change (radixark/sgl-router-for-miles#24) + the miles consumer (radixark/miles#3952). Stack B is f19dcfb12 with the released gateway and miles. Each served Qwen3.5-9B on one H200 with the production engine flags (TP1, FP8 KV cache, 32K context), behind the miles session server.

• Wire. With the flags, P returns output_top_logprobs_shape [32, 128], no nested rows and empty OpenAI top_logprobs: 49 KB instead of 453 KB for 32 tokens. The released gateway drops the flags, so B returns nested rows; a patched gateway in front of an unpatched engine also gets nested rows. Deploy order doesn't matter.
• Equivalence. Greedy, 64 tokens: flat and nested from the same engine have identical token ids and logprobs (max difference 0.0), and B and P generate the same tokens and candidates.
• Latency through the session server, median of 8:

setting	B	P	backend body B → P
k=0, 160 tokens	0.83 s	0.83 s	22 KB → 22 KB
k=128, 160 tokens	1.55 s	0.84 s	2.38 MB → 0.24 MB
k=0, 1000 tokens	5.08 s	5.06 s	133 KB → 133 KB
k=128, 1000 tokens	8.91 s	5.16 s	15.1 MB → 1.5 MB

• Concurrency. 32 concurrent k=128 sessions of ~1000 tokens: B takes 130–134 s, with the TokenizerManager process at ~100% CPU returning one ~15 MB response every 3–4 s while the engine-side e2e median is 37–41 s. P takes 7–10 s.
• Training. Three short miles score-centering runs at k=128 consumed the arrays: every saved rollout has width-128 arrays with no NaN and passes validation. The agent was a stand-in that keeps the multi-turn tool-call traffic but doesn't execute commands.

Checklist

• Format your code according to the Format code with pre-commit. (isort 7.0.0, ruff 0.15.1, codespell 2.4.1, registered-test lint)
• Add unit tests according to the Run and add unit tests.
  • test_flat_raw_top_logprobs.py: validation and batch propagation, no output top logprobs before finish, flat replacing nested on finish, b64 round trip keeping -inf, ragged-row fallback, and convert_logprob_style forwarding finished.
  • test_serving_chat.py: the return_meta_info requirement and the request conversion.
  • Results on main (c0df52be2): test_flat_raw_top_logprobs.py, test_json_response.py, test_serving_chat.py and test_serving_completions.py give 218 passed vs 208 on main; test/registered/unit/managers, test/registered/unit/entrypoints/openai and test/registered/unit/utils give the same failure set as main on a CPU-only host (60 failed and 2 errors from the missing GPU), with 10 more passing. After rebasing onto 87845cda2: test/registered/unit/api (incl. test_parity.py), test_generate_api_contract.py and the four files above give 263 passed.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ⏳ Run #38476261696
Latest PR Test (Extra): ❌ Run #38476261509
Latest PR Test (AMD ROCm 10): ⏳ Run #38476261874

[ByronHsu]

Non-blocking efficiency suggestion for _build_flat_output_top_logprobs_fields: when return_b64=False, the current path converts the Python rows to float32/int32 NumPy arrays, then calls .reshape(-1).tolist() to create Python number objects again before JSON encoding:

Python rows -> NumPy arrays -> new Python flat lists -> JSON

For plain JSON, could we share the rectangular-row/null-prefix validation and flatten the original rows directly? That would create flat lists referencing the existing scalar objects:

Python rows -> flat lists -> JSON

At 1,000 positions x 128 candidates, this could avoid about 1 MB of temporary numeric buffers and recreating up to 256,000 Python scalar objects. The base64 path should keep the NumPy arrays for packing float32/int32 bytes. If the intended contract also requires dtype conversion in plain JSON mode, that should be preserved or documented before changing this.

This is a small follow-up optimization, not a correctness issue; the larger win here is already skipping candidate text and duplicate objects.

[HJSang]

@ByronHsu thanks! Good catch!

[JustinTong0323]

/rerun-failed-ci

[JustinTong0323]

/rerun-failed-ci

[hnyls2002]

/rerun-test test_srt_endpoint.py test_openai_server.py test_skip_tokenizer_init.py test_openai_completion_rust.py test_io_struct.py

[github-actions]

🚀 1-gpu-5090 (5 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py
cd test/ && python3 registered/openai_server/basic/test_openai_server.py
cd test/ && python3 registered/tokenizer/test_skip_tokenizer_init.py
cd test/ && python3 registered/openai_server/basic/test_openai_completion_rust.py
cd test/ && python3 registered/unit/managers/test_io_struct.py