[hnyls2002]

Beam search as a per-request feature: sampling_params.beam_width = k runs a
k-wide beam and returns the top-n sequences, n defaulting to 1 as in HF and
the OpenAI API (n <= k; n > 1 returns that many candidates instead of
parallel samples). No server flag; beam and ordinary requests share the same
batches and memory pools.

Results

Qwen/Qwen3-0.6B-Base, 1x H200, beam_width = 5000. End-to-end latency (ms),
mean / p50 / p90:

output tokens	sync	overlap
3	244.6 / 158.0 / 469.0	259.1 / 176.6 / 470.4
16	927.7 / 1028.8 / 1120.6	903.9 / 957.6 / 1361.7
32	1690.7 / 1788.6 / 1827.3	1650.9 / 1716.8 / 2003.0
64	3511.0 / 3554.1 / 3697.7	3303.3 / 3277.3 / 3506.5
128	8087.3 / 8267.2 / 8655.6	7067.0 / 7164.5 / 7201.3

sglang:fwd_occupancy median under saturated load is 43.7-50.1%, at 67-87%
whole-device utilization.

Repro

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B-Base --port 30000 \
  --dtype bfloat16 --attention-backend flashinfer \
  --mem-fraction-static 0.4 --max-running-requests 20004 \
  --context-length 4096 --random-seed 0 \
  --disable-radix-cache --disable-chunked-prefix-cache
  # add --disable-overlap-schedule for the sync column

Drive concurrent /generate calls with {"beam_width": 5000, "n": 5000},
sweeping max_new_tokens. Per row, (max_new_tokens, num_requests, qps, concurrency): (3, 23, 0.8, 4), (16, 10, 0.3, 2), (32, 8, 0.25, 2),
(64, 6, 0.2, 2), (128, 4, 0.15, 2).

For the occupancy row, add SGLANG_ENABLE_METRICS_DEVICE_TIMER=1,
--enable-metrics --decode-log-interval 2, and take the median of
sglang:fwd_occupancy. --decode-log-interval 1 reports NaN: the window
resets every batch and never reaches the branch that computes the ratio.

Verification

• Unit tests (beam_search, managers, mem_cache): 2062 passed.
• HF parity (test/manual/beam_search/test_beam_parity.py): sequence-set
  overlap 8/10 at k = 10.
• Load (test/manual/beam_search/test_beam_search_load.py): mixed widths 2-100,
  100 requests at 10 QPS, p50/p90/p99 = 53/58/175 ms; 3200-wide fan-out passes.
• Beam mixed with ordinary penalized requests in the same batches: 36/36.
• Sync and overlap agree digest-for-digest, with
  SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_BUSY=1 silent throughout.
• Ordinary-path regression: few_shot_gsm8k 200q greedy 0.470 unchanged.

Not supported yet

Rejected at admission with an explicit error: speculative decoding, PD
disaggregation, dp attention, pipeline parallelism, page_size > 1,
hierarchical cache, SWA/mamba hybrid caches, hisparse, encoder-decoder,
diffusion LLM, LoRA, sessions, constrained decoding, stop strings/regex,
min_new_tokens, return_logprob, return_hidden_states, and
sampling-mask/routed-experts returns. Beam requests are also excluded from
mixed-chunk prefill batches.

Retraction. A beam group cannot be retracted: its member rows alias the
leader's prompt KV, so rolling back part of the group corrupts it. Under KV
pressure ordinary requests are retracted first, and if that is not enough the
whole group is aborted with HTTP 500 instead of being requeued. Admission
reserves the group's req_to_token rows but still budgets decode KV at one
token per step, so a large beam_width can reach this path under load.

---

CI States

Latest PR Test (Base): ✅ Run #33016852973
Latest PR Test (Extra): ✅ Run #33026857281
Latest PR Test (AMD ROCm 7.2): 🚫 Run #33016853050

[vedantjh2]

Thank you!! @hnyls2002

[blacker521]

Introducing FlashRec: a wide-beam inference engine for generative recommendation

Hi all,

Today we’re excited to introduce FlashRec, an open-source inference engine for generative recommendation.

https://github.com/sohu-mptc/FlashRec

General-purpose LLM engines (vLLM, SGLang, TensorRT-LLM) are optimized for long, single-path decoding. Generative recommendation is a different shape: a few decode steps, a very wide beam, and every continuation must land in a valid semantic-ID catalog. A wide-beam request occupies the engine; throughput barely scales with concurrency; open-vocabulary decoding leaves ~17–27% of beams illegal.

FlashRec is built for that workload:

• CUDA-graph capture of the decode path, including beam expansion, at widths 50–512+
• A fused SID trie (dense or CSR) so illegal IDs are never candidates
• Scheduling by beam-row slot budget, so rows from different requests share a step
• FP8 by default (W8A8 weights, fp8 KV), OpenAI-compatible /v1/chat/completions

On public OneRec-1.7B (OpenOneRec RecIF-Bench, RTX 5090), FlashRec consistently delivers higher throughput than existing engines, achieving up to 2.0× vs SGLang, 2.2× vs TensorRT-LLM, and 7.2× vs vLLM. On production SoHuRec traffic, up to ~3× vs SGLang. The illegal-SID rate is 0.

FlashRec is fully open-source (Apache-2.0), with a layout following mini-sglang. You can serve a public checkpoint with:

flashrec --serve --model-path ./OneRec-1.7B --port 8000 \
  --beam-width 32 --max-tokens 5 \
  --sid-vocab-file data/catalogs/sid2pid_beamrec_l4.json

We welcome the community to try it, file issues, and contribute. Please feel free to forward.

Thanks,
Chongyang Wang
chongyangwang@sohu-inc.com
sohu-mptc / FlashRec