Most inference engines are named after speed. SGLang is named after a data structure. The project from LMSYS grew out of RadixAttention, the idea that every prompt your users send is a prefix that has probably been seen before — and if you store attention keys and values in a radix tree keyed by token sequence, you can reuse the expensive part of the computation instead of repeating it. The repository behind that idea has since become a full serving system for language, vision-language, and diffusion models, tuned for agentic workloads where the same long context gets requested again and again.

The codebase reads like an operating system for tokens. There is a process trio — tokenizer manager, scheduler, detokenizer — connected by queues. There is a 5,599-line scheduler that forms batches, retracts them under memory pressure, and asks a cache-aware policy which waiting request deserves to go next. There is a separate Rust gateway for routing across replicas, speculative engines that gamble on draft tokens, and a disaggregation layer that lets one fleet of machines do the reading while another does the writing. The README candidly credits its teachers, acknowledging designs and code reused from vLLM, FlashInfer, Guidance, Outlines, LMQL, and LightLLM — a lineage worn openly rather than hidden. All of it is published, readable, and opinionated. As always in this series, what follows is an educational tour of published source code.

From the launcher through the HTTP surface and the scheduler's control plane, down to the radix cache, execution workers, and advanced engines.

Reading the overview from left to right:

• The launcher is a thin console entry that boots the whole server from parsed arguments.
• The HTTP server registers health probes and generation routes in one 2,565-line FastAPI application.
• The OpenAI layer translates chat payloads, tools, and reasoning fields into internal requests.
• The tokenizer manager is the front desk: tokenize, dispatch, stream increments back to clients.
• The scheduler is the 5,599-line heart that forms batches and owns KV memory decisions.
• The TP worker bridges scheduling decisions to tensor-parallel model execution.
• The radix cache matches incoming prompts against stored prefixes to reuse computed attention.
• The EAGLE worker drafts several tokens with a small model and verifies them in one pass.

Why You Need This

First, the prefix cache is the clearest production lesson in this codebase. The radix tree in mem_cache/radix_cache.py inserts every processed prompt as a path, so a second request sharing a system prompt, a document, or a chat history resumes from the matched node instead of recomputing. Eviction is a policy module, evict_policy.py, so forgetting is swappable; the physical pages live in paged pools in memory_pool.py, separated from the tree that indexes them. The tree even supports locking a node while its owner request is in flight, so an in-progress generation cannot have its history evicted underneath it — a concurrency subtlety that plagues every homegrown cache. If you serve any repetitive workload — agents, RAG, coding assistants — this is the pattern that separates a system that feels instant from one that feels expensive.

Second, the scheduler is a complete, readable case study in batched serving. managers/scheduler.py runs an event loop over a waiting queue and a running batch: new requests are scored by schedule_policy.py, which deliberately favors requests whose prefixes already live in the cache; running batches grow until memory or latency budgets bind; and when memory runs short, the scheduler retracts requests and re-queues them rather than crashing. The dataclasses in schedule_batch.py — 3,517 lines of them — carry a request through every one of those states, which makes the whole lifecycle traceable by reading a single file. You will not find a better-documented mental model of throughput-versus-latency tradeoffs anywhere in open source.

The policy module deserves its own paragraph, because it encodes an economic view of serving. schedule_policy.py — 1,530 lines — offers several orderings, and the cache-aware one computes how much of each waiting request’s prefix is already resident, then sorts by that value. Shared system prompts bubble to the front; novel requests wait their turn. This is a real design decision with real consequences: it maximizes tokens per joule of GPU work but asks you to think about fairness, which is exactly the kind of tension engineering blogs usually hide. Here the tension is visible in code, defaults, and comments, and you can adopt or reject it knowingly.

Third, the project shows how to compose advanced techniques without tangling the core. Speculative decoding arrives as workers — EAGLE with a draft model, n-gram matching with none — that plug into the same scheduler loop; the n-gram variant is especially clever, drafting candidate continuations from repetitions already visible in the context, so the gamble costs no extra model at all. Structured output arrives as grammar backends in constrained/, where xgrammar turns a JSON schema into token masks applied at sampling time. Prefill-decode disaggregation arrives as paired coordinators in disaggregation/ — the prefill side alone is 1,590 lines and the decode side 3,061 — letting a prompt-heavy fleet hand computed KV to a generation-heavy fleet across the network. That split sounds exotic until you notice the two phases have opposite resource shapes: prefill is compute-bound reading, decode is memory-bandwidth-bound writing, and no single machine is optimal at both. None of these features required rewriting the engine; each is a strategy the scheduler can select. That is what good extension seams look like.

The detail view: entry internals, the manager trio, execution plumbing, memory internals, advanced engines, and the Rust gateway.

The control-plane trio rewards a close read. tokenizer_manager.py — 3,729 lines — owns client state, fans requests to the scheduler over inter-process queues, and stitches streamed deltas back into coherent responses. The scheduler delegates output tokens to detokenizer_manager.py, a separate process that turns ids into text, so detokenization never stalls the GPU loop. Scheduling itself overlaps with compute through overlap_utils.py: the CPU prepares step N while the GPU still finishes step N-1, hiding latency that would otherwise serialize the whole server. Sampling parameters, stop conditions, and per-request state all cross those process boundaries as plain messages, which means the whole control plane can be profiled with ordinary tools — no exotic instrumentation required. Each process has one job, the queues between them are explicit, and the result is a system you can reason about under load.

Execution is equally layered. model_runner.py loads weights, initializes cache pools, and drives one forward pass; parallel_state.py — 3,101 lines — initializes tensor, expert, data, and stage parallel process groups; and the attention layer every model calls is literally named radix_attention.py after the cache it serves. Backend selection is a registry, attention_registry.py, matching hardware to one of the many attention implementations in layers/attention/, from FlashInfer-style paths to vendor-specific kernels. Logits assembly, vocab-parallel gather, and per-request sampling parameters each get their own module. The layering means a new GPU backend is a registry entry, not a refactor.

The edges of the system are where the ambition shows. entrypoints/engine.py — 1,860 lines — exposes the same runtime as an in-process Python class, so researchers can drive the scheduler from a script instead of HTTP. A complete OpenAI-compatible surface in entrypoints/openai/ means existing clients migrate unchanged, including tool calls parsed by function_call_parser.py from the assorted dialects models emit. Server configuration is centralized in server_args.py, 773 lines of flags covering parallelism, cache sizing, and backend selection — one file that doubles as a feature inventory of the entire engine. And for clusters, sgl-model-gateway/ is a separate Rust service with cache-aware routing in policies/cache_aware.rs — it prefers replicas that already hold a conversation’s prefix, the same insight as the in-process cache, applied at load-balancer scale. From one GPU to a fleet, the prefix is money in the bank at every layer.

From Install to First Query

The README’s own on-ramp is two commands:

docker pull lmsysorg/sglang:latest
uv pip install --prerelease=allow sglang

Launch a model with the bundled server entry, point any OpenAI client at the /v1/chat/completions route, and watch the logs: the scheduler will report batch composition and cache hit rates as requests arrive. Send the same long document twice and the second request finishes its preface almost immediately — that is the radix tree paying rent in real time. The development path is equally direct — the repository is installable editable from the python/ directory, and the docs recommend starting from the project’s dev container so kernels and dependencies are prebuilt. Because the whole control plane is ordinary Python, the fastest way to learn this codebase is to run a small model locally, set a breakpoint inside the scheduler loop, and watch requests become batches.

Honest limits: this is a serving system, not a training framework — RL libraries and fine-tuning tools integrate with it from outside. The breadth of hardware support, from NVIDIA and AMD GPUs to TPUs, NPUs, and Apple silicon, means backend quality genuinely varies by platform, and the cutting-edge paths carry the rough edges you would expect. The scheduler’s cache-first bias is a deliberate trade: workloads with no shared prefixes will see less benefit from the machinery this engine is named for, and mixed tenants can influence each other through eviction pressure just as they would in any shared cache. The codebase itself moves fast, so file sizes and module boundaries cited here are a snapshot of a moving target. And advanced engines like disaggregation assume real infrastructure, not a laptop. None of that dims the lesson: pick one computation to never repeat, build a data structure around it, and let every layer above — policy, scheduling, routing — compound the savings.

Enjoyed this post? Never miss out on future posts by following us