Local LLM serving has three names developers throw around in 2026: Ollama for simplicity, vLLM for throughput, and increasingly, SGLang for everything in between. If you have tried to run a quantized model on your own GPU this month and hit a wall of conflicting install instructions, dead PyPI wheels, or a server that silently drops half your prompt context, this tutorial walks through the fix end to end. By the close of these 13 steps you will have SGLang installed, a model served behind an OpenAI-compatible API, quantization enabled to cut VRAM use, and a small working project that calls your local server like you would call GPT-6.1 Sol or DeepSeek.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why SGLang Is Becoming the Go-To Local LLM Serving Engine in 2026
Demand for self-hosted inference has not slowed down. Stack Overflow’s 2025 Developer Survey found that 51% of professional developers now use AI tools daily, and separate IDC tracking showed organizations raised spending on AI compute and storage infrastructure by 166% year over year in the second quarter of 2025, reaching $82.0 billion, according to IDC’s Worldwide Quarterly Artificial Intelligence Infrastructure Tracker. That spending has to land somewhere, and a growing share of it is landing on local and self-managed GPU boxes rather than pure API bills.
SGLang sits in that gap. It started as a research project tied to the LMSYS group at Stanford (the same team behind Chatbot Arena) and is now maintained independently under the sgl-project organization on GitHub. As of October 18, 2026, the project carries roughly 36,085 GitHub stars in a snapshot taken September 17, 2026, tracked by Local Model Watch’s engine overview, and its most recent tagged release, v0.5.21 from October 2, 2026, pulled in 779 pull requests from 227 contributors, 55 of them brand new to the project, per the release notes published on SGLang’s own site. A newer package, sglang 0.5.22, appeared on PyPI on October 16, 2026, though it had not yet been mirrored as a formal GitHub release tag at the time of writing. NVIDIA now maintains its own parallel release notes for SGLang, which gives a sense of how far the engine has spread into enterprise GPU stacks beyond the original research use case.
What makes SGLang worth the extra setup time instead of just running Ollama is its focus on prefix caching, structured generation, and request scheduling for agentic and tool-calling workloads. Ollama still wins on sheer convenience for a single chat session, and vLLM remains the default pick for high-throughput batch serving, a tradeoff we broke down in our vLLM vs Ollama comparison. SGLang’s pitch is different: it targets the messy middle where you need both decent throughput and features like constrained JSON output, scoring APIs, and multi-turn agent loops that reuse the same prompt prefix over and over. That reuse is exactly what SGLang’s RadixAttention caching was built to exploit, and the v0.5.21 release moved that prefix cache onto a Rust core by default, which the project reports as a meaningful speed gain on repeated-context workloads.
The agent angle is not an abstract selling point. Stack Overflow’s 2025 Developer Survey found that 31% of developers already use AI agents in their day-to-day work, according to the Stack Overflow Technology section of that survey, and agent loops are precisely the pattern that punishes a naive serving setup. Every tool call, every retrieved document, and every retry resends the same system instructions and the same tool schema. A server that recomputes attention over that repeated text from scratch on every single call wastes GPU cycles that a prefix-cache-aware engine like SGLang simply skips. That is the practical reason a growing number of self-hosted agent projects are reaching for SGLang instead of a plain chat server, even though it takes a few more install steps to get running than Ollama does.
SGLang vs vLLM vs Ollama: Where It Actually Fits
Before installing anything, it helps to see the three engines side by side. None of these numbers come from a neutral, identically-configured benchmark lab; they are the figures each project itself reports in its own release notes, so treat them as directional rather than a scientific head-to-head.
| Engine | Latest version (Oct 18, 2026) | Release date | Primary strength | Install command |
|---|---|---|---|---|
| Ollama | v0.40.6 | Oct 16, 2026 | Fastest path to a running local model | curl -fsSL https://ollama.com/install.sh | sh |
| vLLM | v0.31.0 | Oct 5, 2026 | High-throughput batch and production serving | pip install vllm |
| SGLang | 0.5.22 (PyPI) / v0.5.21 (tagged release) | Oct 16, 2026 / Oct 2, 2026 | Prefix caching, structured output, agent workloads | uv pip install –prerelease=allow sglang |
The release notes for SGLang v0.5.21 cite a 22% faster time-to-first-token for DeepSeek-V4.1 on long prompts and a 20.6% higher prefill throughput for Kimi K3 under prefill/decode disaggregated serving. Those are real, documented gains for those specific model and workload combinations, not a universal claim that SGLang beats vLLM or Ollama in every scenario. If your workload is a single developer chatting with one local model, Ollama’s setup covered in our Ollama self-hosting guide is still the fastest route. If you are serving dozens of concurrent agent sessions that keep reusing the same system prompt or tool schema, SGLang’s prefix cache and new /v1/decisions and /v1/score APIs start to pay for the extra install complexity.
How SGLang Fits Into a Local AI Stack
It helps to picture the full stack before touching a terminal. A local LLM setup usually has three layers: a serving engine that loads model weights onto a GPU and exposes an API, a model (or several, if you are running a router), and whatever application code talks to that API. Ollama tends to blur the first two layers together, since ollama run model_name both downloads and serves in one step. SGLang keeps them more separate on purpose: you choose the exact checkpoint, the exact quantization format, and the exact parallelism strategy, which is more typing but gives you control that matters once you move past a single personal GPU.
That separation also means SGLang slots cleanly underneath frameworks you may already use. Because its API speaks the same OpenAI chat completions schema that GPT-6.1 Sol, Claude, and Gemini endpoints use, swapping a cloud model for a locally served one in LangChain, LlamaIndex, or a custom agent loop is usually a one-line change: point the base_url at your own server and drop the API key requirement. The model itself does not need to know or care that it is being called through SGLang rather than a managed API, which is the entire point of standardizing on an OpenAI-compatible interface in the first place.
Where this gets interesting is cost. A team running a steady stream of agent calls against a hosted API pays per token, every time, for every retry. The same workload running against a self-hosted SGLang server on owned or rented GPU hardware costs electricity and amortized hardware, full stop, regardless of how many times an agent loop retries a failed tool call or re-reads a long document. That math does not favor self-hosting for every team, especially at low volume, but for anyone running agent workloads at meaningful scale, it is the calculation driving a lot of the renewed interest in local serving engines this year.
Prerequisites: Hardware, Software, and Exact Versions You Need
SGLang’s own documentation lists support for NVIDIA CUDA GPUs, AMD ROCm GPUs, CPUs, TPUs, NPUs, and Apple Metal, which is a wider hardware net than most local-inference tools throw. For this walkthrough we are targeting the most common setup: a single NVIDIA GPU on Linux. Confirm each line below before you start the install steps.
| Component | Minimum | Recommended for this tutorial |
|---|---|---|
| Operating system | Ubuntu 22.04 or later | Ubuntu 24.04 LTS |
| Python | 3.10 | 3.12 (the version SGLang’s newest prebuilt wheel targets on PyPI) |
| GPU | 8GB VRAM with AWQ INT4 quantization | 16GB+ VRAM for an unquantized 8B model |
| NVIDIA driver / CUDA | CUDA 12.x compatible driver | Latest driver branch supporting CUDA 12 |
| Package manager | pip | uv (the installer SGLang’s docs recommend) |
| Container runtime (optional) | Docker 24+ | Docker with NVIDIA Container Toolkit for the lmsysorg/sglang image |
| Disk space | 20GB free | 60GB+ if you plan to cache multiple model checkpoints |
You will also want a free Hugging Face account if you plan to pull a gated checkpoint such as Meta-Llama-3.1-8B-Instruct, since SGLang downloads model weights the same way most serving engines do, straight from the Hugging Face Hub cache.
Step 1-2: Prepare Your Environment and Install uv
Start clean. Mixing SGLang into a Python environment that already has conflicting torch or CUDA toolkit versions is the single most common source of install failures reported in SGLang’s GitHub issues. Create an isolated virtual environment first, then install uv, the package manager the project’s own quickstart now recommends over plain pip.
# Step 1: create and activate an isolated environment
python3.12 -m venv ~/sglang-env
source ~/sglang-env/bin/activate
# Step 2: install uv, the recommended installer
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/envuv exists because SGLang ships with a large, fast-moving dependency tree (torch, flashinfer, transformers, and several CUDA-specific kernels). Pip can resolve that tree, but it is noticeably slower and more prone to version conflicts than uv’s resolver. If you are on a machine where you cannot install uv globally, a plain pip install still works for every command in this tutorial; just swap “uv pip install” for “pip install” throughout.
Step 3-4: Install SGLang and Verify the Build
With uv in place, install SGLang itself. Because the project ships fast and some of its newest kernel optimizations land as pre-release builds, the documented install command explicitly allows pre-release versions.
# Step 3: install SGLang (pulls in torch, flashinfer, and serving deps)
uv pip install --prerelease=allow sglang
# Step 4: verify the install and check the version
python3 -c "import sglang; print(sglang.__version__)"Expect output close to this, depending on which package snapshot uv resolves at install time:
0.5.22If this command errors out with a missing CUDA symbol or a torch mismatch, do not try to patch individual packages. Delete the virtual environment, confirm your NVIDIA driver with nvidia-smi, and recreate the environment from Step 1. This is faster than chasing dependency conflicts one at a time, and it mirrors the recommendation several SGLang contributors give in the project’s GitHub discussions.
Step 5-6: Download a Model and Launch Your First Server
SGLang does not require a separate download step; pointing launch_server at a Hugging Face model ID triggers the download automatically and caches it under ~/.cache/huggingface. For a first run, an 8-billion parameter instruct model is large enough to be useful and small enough to fit comfortably on a single consumer GPU.
# Step 5: log into Hugging Face if the model is gated
huggingface-cli login
# Step 6: launch the server (classic form)
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 \
--host 0.0.0.0
# The newer CLI form documented in SGLang's current quickstart does the same thing:
sglang serve \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000The first launch downloads several gigabytes of model weights, so expect a delay proportional to your connection speed. Once loading finishes, the terminal prints a line confirming the server is listening, something close to:
[Scheduler] Capture cuda graph begin.
[Scheduler] max_total_num_tokens=163840
Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)Leave this process running in its own terminal tab or behind a tool like tmux. Every later step in this tutorial sends requests to this same server on port 30000.
Step 7-8: Test the OpenAI-Compatible API
SGLang exposes an OpenAI-compatible endpoint at /v1 on the port you launched with, which means any code you already wrote against the official OpenAI SDK or against something like the vLLM setup covered in our earlier tutorial will mostly work unchanged, just pointed at a different base URL.
# Step 7: send a raw curl request to confirm the server responds
curl -s http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "In one sentence, what is RadixAttention?"}],
"max_tokens": 80
}'
# Step 8: the same call from Python using the openai package, pointed at your local server
python3 - <<'PY'
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="not-needed")
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "In one sentence, what is RadixAttention?"}],
max_tokens=80,
)
print(response.choices[0].message.content)
PYA healthy response looks like a normal OpenAI chat completion payload, with choices, a finish_reason, and usage token counts, just served from your own machine instead of a billed API call. If you get a connection refused error here, jump ahead to the troubleshooting section below before changing anything else.
Step 9-10: Add Quantization to Cut VRAM Use
Running a full-precision 8B model comfortably needs 16GB or more of VRAM. Quantization is how you fit larger models, or the same model on smaller GPUs, and it is the same tradeoff we covered in detail in our dedicated GGUF quantization tutorial, just applied through SGLang's own flags instead of llama.cpp's conversion scripts. SGLang's current documentation lists support for AWQ (including AWQ INT4), GPTQ, FP8, plain INT4, W8A8 INT8, and ModelOpt-based quantization, along with online dynamic quantization for models that were not pre-quantized.
| Format | Typical VRAM savings | Launch flag | Best for |
|---|---|---|---|
| AWQ INT4 | ~65-75% smaller than FP16 | Load a pre-quantized AWQ checkpoint directly | Fitting 8B-13B models on 8-12GB GPUs |
| FP8 | ~50% smaller than FP16 | --quantization fp8 | Newer GPUs with native FP8 support |
| GPTQ | ~65-75% smaller than FP16 | Load a pre-quantized GPTQ checkpoint directly | Models with an existing GPTQ build on Hugging Face |
| W8A8 INT8 | ~50% smaller than FP16 | --quantization w8a8_int8 | CPU-backed serving, per SGLang's CPU server docs |
# Step 9: serve a pre-quantized AWQ INT4 checkpoint directly, no extra flag needed
python3 -m sglang.launch_server \
--model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
--port 30000 \
--host 0.0.0.0
# Step 10: run FP8 quantization on a full-precision checkpoint instead
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--quantization fp8 \
--port 30000 \
--host 0.0.0.0SGLang reads the quantization method from the model's own config when you load a checkpoint that was already quantized upstream, so in most cases you do not pass a --quantization flag at all for formats like AWQ or GPTQ; it only matters when you want SGLang to quantize a full-precision model on the fly, which is what the --quantization fp8 flag does. A useful recent example of how far quantization can stretch hardware: OrcaRouter's OrcaSAQ-2 27B model, released September 28, 2026 under an Apache 2.0 license, compressed a roughly 54GB BF16 checkpoint of Alibaba's Qwen3.8-27B down to about 12.3GB using a mixed-precision scheme, small enough to run on a single 16GB GPU.
Step 11-12: Scale With Tensor Parallelism and Prefix Caching
Once a single-GPU server works, the next question is usually "what if the model does not fit on one card, or what if I need more throughput." SGLang's v0.5.21 release specifically called out more accurate behavior under pipeline parallelism (PP), data-parallel (DP) attention, and context parallelism (CP), with SGLang itself now handling layer communication across those modes rather than leaving it to lower-level libraries.
# Step 11: split a larger model across two GPUs with tensor parallelism
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-70B-Instruct \
--tp-size 2 \
--port 30000 \
--host 0.0.0.0
# Step 12: the prefix cache is on by default (Rust-backed since v0.5.21),
# but you can confirm it is active in the server logs on startup
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 \
--host 0.0.0.0 \
--log-level infoPrefix caching matters most when many requests share the start of a prompt, which is exactly the pattern in agent loops that resend a long system prompt or tool schema on every turn. That is also why v0.5.21 added dynamic switching between prefill and decode roles on running instances without a restart, a feature aimed squarely at production deployments where you cannot afford downtime to rebalance load between the "reading the prompt" and "generating tokens" phases of serving.
Step 13: Containerize SGLang With Docker for Production
For anything beyond a personal dev box, running SGLang inside a container makes upgrades and rollbacks far less painful than managing a bare virtual environment on a shared server. SGLang's maintainers publish prebuilt images under the lmsysorg/sglang namespace on Docker Hub.
# Step 13: pull the prebuilt image and run it with GPU access
docker pull lmsysorg/sglang:latest
docker run --gpus all -it --rm \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000
# CPU-only build, for machines without a supported GPU
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
docker build -t sglang-cpu:latest -f xeon.Dockerfile .Mounting your Hugging Face cache directory into the container, as shown above, avoids re-downloading multi-gigabyte model weights every time the container restarts. This is the single biggest time-saver for anyone iterating on a production SGLang deployment, since weight downloads otherwise dominate restart time.
Measuring Your Server's Throughput
Before trusting any of the percentage gains quoted in SGLang's release notes, measure your own setup. Tokens-per-second depends heavily on your specific GPU, model size, quantization format, and prompt length, so a quick local benchmark is worth more than any published number for deciding whether your configuration is actually fast enough for your use case.
# benchmark_server.py
import time
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="not-needed")
PROMPT = "Write a 200-word explanation of how a hash table resolves collisions."
def run_once():
start = time.time()
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": PROMPT}],
max_tokens=256,
)
elapsed = time.time() - start
tokens = response.usage.completion_tokens
print(f"{tokens} tokens in {elapsed:.2f}s -> {tokens / elapsed:.1f} tok/s")
for _ in range(5):
run_once()Run this five-request loop a couple of times in a row. The first call after a cold server start will usually be slower, since it includes CUDA graph capture and initial cache warmup; the following calls give a more realistic steady-state number. If your tokens-per-second figure looks far lower than you expected for your GPU class, go back to the quantization section above before assuming your hardware is the bottleneck, since an unquantized model on a GPU that is close to its VRAM limit will frequently fall back to slower memory paths without printing an explicit warning.
Build a Complete Working Project: A Local Code-Review Assistant
With the server running, here is a small, complete project that ties everything above together: a command-line tool that reads a diff and asks your locally served model for a structured review, using SGLang's OpenAI-compatible endpoint exactly the way you would call a hosted model.
# code_review_bot.py
import sys
from openai import OpenAI
MODEL = "meta-llama/Meta-Llama-3.1-8B-Instruct"
SYSTEM_PROMPT = (
"You are a terse code reviewer. Given a diff, list up to 3 concrete "
"issues as a numbered list. If the diff looks fine, say so in one line."
)
def review_diff(diff_text: str) -> str:
client = OpenAI(base_url="http://localhost:30000/v1", api_key="not-needed")
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": diff_text},
],
max_tokens=300,
temperature=0.2,
)
return response.choices[0].message.content
if __name__ == "__main__":
diff = sys.stdin.read()
print(review_diff(diff))Run it by piping a git diff straight in:
git diff HEAD~1 | python3 code_review_bot.pyBecause this calls the same SYSTEM_PROMPT and MODEL on every invocation, it is also a good illustration of why prefix caching helps in practice: every single call to this script shares the same long system prompt, which is exactly the repeated-prefix pattern SGLang's Rust-backed cache is built to speed up. Swap the model path for the AWQ INT4 checkpoint from Step 9 if you want to run this on a smaller GPU, or point MODEL at any other Hugging Face checkpoint your server is currently loaded with.
5 Common Pitfalls When Setting Up SGLang
- Installing into a dirty environment. Running uv pip install --prerelease=allow sglang inside an environment that already has a different torch or CUDA build installed is the most common cause of import errors. Always start from a fresh virtual environment.
- Forgetting the --host 0.0.0.0 flag. Without it, SGLang binds to localhost only, which silently blocks any request coming from another machine, a Docker container, or even some remote-desktop setups.
- Assuming every quantization format needs a flag. Pre-quantized AWQ and GPTQ checkpoints read their quantization method straight from the model config. Adding --quantization on top of an already-quantized model can cause a mismatch error instead of helping.
- Not checking VRAM before picking a model size. A 70B model will not fit on a single consumer GPU even with aggressive quantization. Check nvidia-smi before you pick a checkpoint, not after the server crashes mid-load.
- Treating release-note benchmarks as universal truth. The 22% and 20.6% throughput gains cited in SGLang's own v0.5.21 notes are tied to specific models (DeepSeek-V4.1, Kimi K3) and specific serving modes. Your workload and model may see very different numbers.
Troubleshooting: 8 Issues You Will Likely Hit
| Symptom | Likely cause | Fix |
|---|---|---|
| Connection refused on port 30000 | Server still loading, or bound to localhost only | Wait for the Uvicorn running on line, confirm --host 0.0.0.0 was set |
| CUDA out of memory on launch | Model too large for available VRAM | Switch to an AWQ INT4 or FP8 checkpoint, or reduce --tp-size concurrency |
| ImportError on sglang.launch_server | Conflicting torch version from a prior install | Delete the virtual environment and reinstall from a fresh venv |
| Model download stuck or 401 error | Gated model, no Hugging Face token set | Run huggingface-cli login before launching the server |
| Responses stop mid-sentence | max_tokens set too low for the request | Raise max_tokens in your client call, not on the server itself |
| Quantized model loads but produces garbage output | Wrong quantization method assumed by SGLang | Confirm the checkpoint's config.json lists the correct quant method |
| Docker container cannot see the GPU | NVIDIA Container Toolkit missing or misconfigured | Reinstall the toolkit and confirm docker run --gpus all works on a test image first |
| Slow first request after startup | CUDA graph capture running on first launch | This is expected behavior; subsequent requests are faster once capture finishes |
Advanced Tips: Structured Generation, Scoring, and Multi-GPU Tuning
Once the basic setup is stable, SGLang's newer APIs are where the engine starts to differentiate itself from a plain chat server. The v0.5.21 release added a /v1/score endpoint for scoring multiple candidate completions in a single request, useful for reranking retrieved documents or comparing several tool-call options without round-tripping the model multiple times. It also added a /v1/decisions API aimed at low-latency classification and yes/no scoring tasks, which is a meaningfully cheaper pattern than generating a full free-text response just to parse out a label.
For structured output specifically, SGLang supports grammar- and schema-guided generation, which constrains the model's token choices so the output is guaranteed to match a JSON schema or regular grammar rather than hoping the model follows instructions. This matters for the same reason speculative decoding matters for raw speed, a technique we cover separately in our speculative decoding setup guide: both are ways of squeezing more reliable, faster output out of the same underlying model weights without retraining anything.
On the multi-GPU side, pay attention to which parallelism mode matches your bottleneck. Tensor parallelism (--tp-size) splits a single model's layers across GPUs and helps when the model itself does not fit on one card. Data parallelism instead runs full copies of the model on each GPU and routes separate requests to each, which helps raw request throughput more than it helps fitting a large model. SGLang's v0.5.21 notes specifically called out improved accuracy under combined PP, DP-attention, and CP configurations, so if you are running a multi-node setup, pulling the latest release matters more than it would for a single-GPU deployment.
Here is what grammar-guided structured output looks like in practice, forcing the model to return valid JSON matching a fixed schema instead of free-form text you would otherwise have to parse with a regular expression and hope for the best:
# structured_output.py
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="not-needed")
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Extract the ticket priority and summary from: 'Server down, revenue impact, page oncall now.'"}],
extra_body={
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "ticket",
"schema": {
"type": "object",
"properties": {
"priority": {"type": "string", "enum": ["low", "medium", "high", "critical"]},
"summary": {"type": "string"}
},
"required": ["priority", "summary"]
}
}
}
},
)
print(response.choices[0].message.content)With the schema enforced, the output is guaranteed to be parseable JSON on every call, which removes an entire category of bugs from any pipeline that otherwise depends on a model reliably following a "please respond in JSON" instruction. This is the same constrained-decoding idea that underlies tool-calling support in hosted APIs, just running on weights you control.
Which Models Actually Make Sense to Serve Right Now
SGLang will happily load most Hugging Face checkpoints, but not every model is worth the disk space and VRAM for a local setup in October 2026. A few recent releases are worth knowing about specifically because they were built with exactly this use case in mind. Liquid AI shipped LFM2.5-24B-A2B on October 13, 2026, a model with 24 billion total parameters but only about 2 billion active at inference time, which reportedly ran at 184 tokens per second on an Apple M5 Max in one independent comparison. That active-parameter design is a mixture-of-experts approach, and it is a good fit for anyone who wants a large model's knowledge without paying the full compute cost of a dense 24B model on every token.
For GPU-constrained setups specifically, OrcaRouter's OrcaSAQ-2 27B, released September 28, 2026 under an Apache 2.0 license, is worth testing against your own hardware. It is a mixed-precision quantized build of Alibaba's Qwen3.8-27B that shrank a roughly 54GB BF16 checkpoint down to about 12.3GB, small enough to serve from a single 16GB consumer GPU. Loading either of these through SGLang follows the exact same launch_server pattern covered in Steps 5 and 6, just with a different --model-path value; neither needs special handling beyond what this tutorial already covers.
If you are choosing between several candidate models for a given task, do not assume parameter count alone predicts quality for your workload. Independent comparisons have found that LFM2.5-24B-A2B beats larger rivals like Qwen3.6-35B-A3B on tool-use benchmarks while losing to that same model on SWE-Bench Verified, a coding-specific benchmark. The practical takeaway is to benchmark the two or three candidates that fit your VRAM budget on your own representative prompts, using the throughput script from the previous section, rather than picking a model purely off a leaderboard number that may not reflect your actual use case.
Monitoring and Keeping SGLang Updated
SGLang ships fast, with 779 pull requests landing in the v0.5.21 release alone. That pace is good for features but means pinning a version in any production deployment is worth doing deliberately rather than always tracking the newest PyPI build.
# Pin to a specific known-good version instead of always pulling latest
uv pip install --prerelease=allow "sglang==0.5.21"
# Check for new releases before upgrading a production server
pip index versions sglangFor day-to-day health checks, the server's startup logs print the max_total_num_tokens figure for your loaded model, which is a quick way to confirm your KV cache budget before you start load-testing. If that number looks lower than expected for your GPU, it usually means another process is holding VRAM, which nvidia-smi will reveal immediately.
It is also worth setting up a basic health-check endpoint call in whatever process manager or orchestration layer restarts your server, since a crashed SGLang process does not automatically come back on its own. A simple cron job or systemd unit that curls the server's /health path (or falls back to a lightweight chat completion request if no dedicated health route is configured) and restarts the service on failure closes the gap between "it worked when I tested it" and "it stays up overnight."
For a broader look at how the open-source model ecosystem SGLang serves is evolving, our AI and machine learning coverage tracks new releases from DeepSeek, Liquid AI, and the rest of the open-weight field as they ship.
Frequently Asked Questions
Is SGLang free and open source?
Yes. SGLang is maintained by the sgl-project organization on GitHub and is free to install and run on your own hardware, with no licensing fee tied to the engine itself.
Does SGLang work without an NVIDIA GPU?
SGLang's documentation lists support for AMD ROCm GPUs, CPUs, TPUs, NPUs, and Apple Metal in addition to NVIDIA CUDA, including a dedicated CPU server build with its own Docker image. Performance on non-NVIDIA hardware will generally be slower than on a comparable CUDA setup.
How much VRAM do I need to run an 8B model with SGLang?
Roughly 16GB or more for a full-precision 8B model. With AWQ INT4 quantization, that same model class can run comfortably on GPUs with 8GB of VRAM.
Is SGLang faster than vLLM?
SGLang's own v0.5.21 release notes report specific gains, such as a 22% faster first token for DeepSeek-V4.1 on long prompts, but these figures are not from a neutral side-by-side benchmark against vLLM. Which engine is faster depends heavily on your model, hardware, and whether your workload benefits from prefix caching.
Can I use SGLang with models other than Llama?
Yes. SGLang's release notes list support for a wide range of LLM and vision-language models, including models like DeepSeek-V4.1 Flash and GigaChat 3.5, alongside standard Llama-family checkpoints from Hugging Face.
What is the difference between the classic launch_server command and the newer sglang serve command?
Both launch the same underlying server. python3 -m sglang.launch_server is the original invocation documented across most existing tutorials and scripts, while sglang serve is the newer, shorter CLI form shown in SGLang's current quickstart. Either works; pick whichever fits your existing automation.
Do I need Docker to run SGLang?
No. Everything in Steps 1 through 12 of this tutorial runs in a plain Python virtual environment. Docker becomes useful once you move toward a shared server or production deployment where reproducible environments and easy rollbacks matter more than they do on a personal dev machine.
What happened to the Stanford or LMSYS branding on SGLang?
The project's Docker images are still published under the lmsysorg namespace, reflecting its LMSYS research origins, but current project materials and its GitHub organization identify it simply as the SGLang project rather than crediting a single university lab or company as its present owner.
