[littlefatfat]

Motivation

This PR introduces SGLang Simulator, a CPU-based inference simulator for
evaluating SGLang scheduling, latency, throughput, and prefix-cache behavior
without loading model weights or executing model kernels.

It is a current-main rework of the simulator prototype in
#22250, while retaining
SGLang v0.5.16 as the tested compatibility floor.

Large-scale inference configuration studies are expensive to run directly on
accelerators. SGLang Simulator reuses the real SGLang scheduler, radix cache,
hierarchical cache, and request lifecycle, while replacing model-forward
execution with a latency predictor. It supports measured traces, synthetic
workloads, and both in-process and HTTP serving workflows.

What This PR Adds

• Add tools/sglang-simulator as a monorepo component.
• Reuse the SGLang scheduler, cache allocator, radix cache, hierarchical cache,
  and request lifecycle with dummy model weights and a minimal KV allocation.
• Add three forward-latency predictors:
  • AIConfigurator;
  • an sklearn-compatible ML predictor;
  • batch-composition replay.
• Support Random, ShareGPT, and timestamped Autobench workloads.
• Support direct Python execution through SGLangBenchmarkRunner.
• Support server and benchmark-client execution through the HTTP serving path.
• Collect request, latency, throughput, prefix-cache, cache-tier, and predictor
  metrics.
• Add reusable example configurations, workloads, tokenizer assets, and replay
  fixtures.
• Add compatibility checks for SGLang launch and serving APIs.
• Run the simulator regression suite from CPU PR CI when
  tools/sglang-simulator changes.

Simulation Modes

The simulator provides two execution modes:

• OFFLINE: advances a logical clock without sleeping, intended for fast
  configuration and workload studies.
• BLOCKING: sleeps for predicted forward latency and visible L2-to-L1 load
  latency, intended for serving-path and timing-semantics validation.

Replay replaces model-forward latency only. CPU preprocessing and
postprocessing overhead remain enabled by default.

For OFFLINE runs, latency, throughput, and duration should be read from the
server-side simulator metrics rather than benchmark-client wall-clock time.

Accuracy Validation

The following results compare simulator output with measured serving traces.
Latency, throughput, and duration values are absolute percentage errors.
Prefix reuse is reported as absolute percentage-point error.

The SGLang baseline column identifies the version used by the real service
that produced the measurement trace.

Model	SGLang baseline	Hardware	Predictor / coverage	TTFT	TPOT	ITL	Input throughput	Duration	Prefix reuse
Qwen3-8B	v0.5.9	H20	AIConfigurator; no-cache, L1, and L2 cases	2.38–4.15%	1.64–4.05%	1.78–4.07%	1.41–2.40%	1.39–2.33%	0.00–0.04 pp
Qwen3-32B-FP8	v0.5.9	H20	AIConfigurator; no-cache, L1, and L2 cases	2.40–3.05%	0.58–1.11%	0.52–1.02%	0.52–1.13%	0.51–1.12%	0.00–0.04 pp
GLM-5.1-FP8	v0.5.11	GPU	ML; L1 and L2 cases	5.27%	N/A	N/A	1.11%	1.08%	0.028 pp
DeepSeek-V4-Pro	v0.5.11	GPU	ML; L1 and L2 cases	6.12%	N/A	N/A	0.23%	0.22%	0.048 pp
DeepSeek-V4-Flash	v0.5.16	GPU	ML; L1 case	0.786–1.343%	N/A	N/A	0.000212%	0.000212%	0.000 pp
DeepSeek-V4-Pro	v0.5.16	GPU	AIConfigurator; L2, 32–128K traces	1.99–10.17%	N/A	N/A	0.0055–0.0183%	0.0055–0.0183%	0.000–0.011 pp

TPOT and ITL are not applicable to the GLM-5.1-FP8,
DeepSeek-V4-Pro, and DeepSeek-V4-Flash validation traces because they use
one-token outputs(Prefill-Decode Disaggregation).

The SGLang v0.5.16 validation additionally covers four measured 32–128K
serving traces. Using AIConfigurator as the forward-latency predictor, the
simulator achieves a mean-TTFT MAPE of 5.89%. Duration and input-throughput
MAPE are both 0.011%, while the maximum absolute prefix-reuse error remains
below 0.011 percentage points.

Prefix-cache accuracy is particularly sensitive to max_total_tokens. This
value determines simulated device KV-cache capacity and also affects
hierarchical host-cache sizing and eviction behavior. For deployment-faithful
comparisons, use the max_total_num_tokens value reported by the real SGLang
server startup log as the simulator's --max-total-tokens value.

---

CI States

Latest PR Test (Base): ⏳ Run #33785893262
Latest PR Test (Extra): 🚫 Run #33830599147
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

[hzh0425]

should fix the lint

[littlefatfat]

/rerun-failed-ci

[littlefatfat]

/rerun-failed-ci

[littlefatfat]

/rerun-failed-ci

[ispobock]

we also need to add a CODEOWNERS entry for tools/sglang-simulator/

[littlefatfat]

/rerun-failed-ci

[hzh0425]

we also need to add a CODEOWNERS entry for tools/sglang-simulator/

let me add in next pr

[hzh0425]

The CI for the simulator has finished running, and most of the tests for `basea`/`baseb` within `base` have already been executed.

This PR introduces a separate, new directory and does not modify any other code.