Our first tour of vLLM walked the main road: requests enter the OpenAI-compatible server, the scheduler batches them, and the executor drives GPU workers that run the model. That post left two guard towers unvisited, and they turn out to be where some of the most interesting engineering lives. The first tower is speculative decoding, the art of having a cheap sidekick guess several tokens at once so the expensive model only has to check them. The second is structured output, the machinery that forces a model to emit JSON, regex-matching text, or any context-free grammar you specify, one constrained token at a time. Both systems are plumbed directly into the scheduling and sampling hot paths, which is what makes them worth reading in the source rather than in the docs. As before, this is an educational walk through the published code of vllm-project/vllm under the Apache-2.0 license; if you want the lay of the land first, the earlier post at /vllm/ covers the serving path end to end.

The repository layout should feel familiar from that first visit: the vllm package holds the Python orchestration, csrc holds the CUDA and C++ kernels, and the v1 subpackage is the engine that matters here. The files in play are compact and readable. The n-gram drafter is 295 lines, the suffix drafter is 102, the rejection sampler is 1,130 lines of mostly Triton kernels, and the structured output manager with its backends is under 2,000 lines in total. Each of these was designed to sit inside a single engine step without adding latency to requests that do not use it, and that constraint explains most of their design decisions.

vLLM's second story: the scheduler carries speculative token reservations and grammar bitmasks into each step, the GPU runner fans drafting out to n-gram, EAGLE, or suffix proposers, the rejection sampler verifies every draft against target logits, and the structured output manager compiles grammars into per-token masks that gate the sampler.

Reading the overview from left to right:

• The continuous batching scheduler at vllm/v1/core/sched/scheduler.py plans each step in schedule() and records already-drafted tokens in scheduled_spec_decode_tokens.
• The GPU model runner at vllm/v1/worker/gpu_model_runner.py runs execute_model() and, after target sampling, calls propose_draft_token_ids() to fan out to the configured drafter.
• The n-gram drafter at vllm/v1/spec_decode/ngram_proposer.py guesses from the request’s own text with zero model weights, while the EAGLE drafter at vllm/v1/spec_decode/eagle.py inherits from SpecDecodeBaseProposer to run a small draft model.
• The suffix drafter at vllm/v1/spec_decode/suffix_decoding.py consults suffix trees built from previous responses, a natural fit for agentic workloads where outputs repeat.
• The RejectionSampler scores every draft token against the target model’s logits and recovers the ones it rejects.
• On the constraint side, the StructuredOutputManager compiles grammars, and the XgrammarBackend turns them into the per-token bitmasks that gate sampling.
• Acceptance statistics flow into SpecDecodingStats, which powers the per-position acceptance-rate logs operators use to tune draft settings.

Choosing a drafter

The entry point is a straight-line dispatch in the model runner, lines 590 through 659. Depending on the speculative configuration, the runner constructs a NgramProposer, an NgramProposerGPU, a DraftModelProposer, a SuffixDecodingProposer, an EagleProposer, one of several model-specific MTP proposers, or a custom class the user registers. Two details in this block repay attention. First, the drafter is only created on the last rank of a sharded deployment, since there is no point drafting tokens on a rank that never samples. Second, the EAGLE family sets a flag that asks the target model to hand over auxiliary hidden states, because EAGLE drafts from the target’s own features rather than from token ids alone. The same block constructs the RejectionSampler, wiring it to the shared sampler so bonus tokens honor the request’s temperature and top-k settings.

The n-gram drafter is the clearest place to see how much mileage you can get without a model. Its constructor asserts that prompt_lookup_min and prompt_lookup_max are configured, pre-allocates a numpy buffer of draft tokens for every possible concurrent sequence, and then does something delightfully pragmatic: it calls its own propose method once at startup on dummy data purely to trigger numba’s JIT compilation, so the first real request does not pay the compile cost. Thread budgeting is equally deliberate; the code caps numba threads at one while tensor-parallel drafting is unimplemented, with a comment noting that the frontend and structured output compilation also compete for CPU threads.

The actual search happens in _find_longest_matched_ngram_and_propose_tokens, and it is a clever inversion. Instead of scanning the context for a matching n-gram, the kernel reverses the token sequence and looks for the longest prefix of the reversed array that matches a suffix, using a KMP-style longest-prefix-suffix array capped at max_ngram to bound memory. The scan tracks the earliest position in the original sequence that achieves the longest match, then drafts the k tokens that follow that match. If the context is shorter than the minimum n-gram, or drafting would overflow the model’s context length, it returns an empty draft. All of this runs inside a numba parallel kernel over the requests in the batch, with a token-count threshold below which threading overhead outweighs the win.

For deployments where copying token ids to the CPU every step is the bottleneck, the 667-line ngram_proposer_gpu.py keeps the token table in device memory and performs the match on the GPU, returning valid draft counts through an async device-to-host copy on a dedicated stream so the scheduler is never blocked.

Drafting with a model: the EAGLE family

EAGLE-style drafters share their machinery in SpecDecodeBaseProposer, an 1,862-line base class that handles everything from batch padding to CUDA graph key management. The heart of the class is propose(). It begins by folding the target model’s hidden states, because EAGLE3 variants consume hidden states from multiple target layers, and the code asserts the combined width matches the drafter’s expectation. Then set_inputs_first_pass arranges the inputs for step zero, where the drafter processes the target’s last hidden state and the token just sampled.

One micro-optimization stands out. For MTP variants that share an indexer across draft steps, the code flips a skip_topk flag: step zero computes its top-k indices and writes them into a shared buffer, steps one and beyond read the same indices instead of recomputing, and a compaction call squeezes them to just the request’s last token. The class also decides how to pad the draft batch for CUDA graph replay, resolves which attention metadata each draft layer needs, and can even detect when the draft model can share the target’s embedding table or lm head outright, saving a second copy of potentially billions of parameters. Sampling the draft itself happens in _sample_draft_tokens, which keeps the last draft probabilities around precisely because the rejection sampler will need them a moment later.

The suffix drafter takes a completely different tack and is worth a paragraph for that reason alone. It imports Arctic Inference’s SuffixDecodingCache and maintains one suffix tree per active request, seeded with the prompt and grown with each sampled token. At propose time it extracts at most max_tree_depth tokens from the end of the context, speculates a variable-length draft whose length is capped by a spec factor over the match length, and stops requests that vanished from the batch. Because agentic workloads call the same tools with nearly identical prompts thousands of times, this drafter can post acceptance rates that fixed-length drafters cannot touch, and the code’s willingness to return drafts of different lengths per request is what the scheduler’s spec-token accounting is built to absorb.

Verification: rejection sampling on the GPU

Drafts are guesses, and the RejectionSampler is the judge. Its forward() receives the draft token ids, optional draft probabilities, and the target model’s logits for every draft position plus one bonus position. The bonus position is sampled first with the request’s own sampler settings, so the token that follows a fully accepted draft is a legitimate sample, not an afterthought. Target logits are cast to float32, passed through the same logits processors the normal path uses, and then apply_sampling_constraints applies temperature and top-k/top-p to the flattened tensor, preserving the guarantee that accepted sequences are distributed exactly as if the target model had generated them alone.

The acceptance step itself, rejection_sample, expands the batch to one row per draft token, generates uniform random numbers, and accepts where the target’s probability of the draft token exceeds the random draw. Rejected positions do not simply truncate the sequence; sample_recovered_tokens draws a rescue token from the residual distribution, the target probabilities minus the draft probabilities clamped at zero, which is what makes the output statistically indistinguishable from undrafted decoding. Everything runs in Triton kernels with dedicated warmup input builders, including a greedy variant that skips randomness entirely for temperature-zero requests.

Back in the scheduler, update_from_output closes the loop. For each request it computes how many of the returned token ids were drafts versus the sampled bonus, derives accepted and rejected counts at lines 2196 through 2218, and rolls num_computed_tokens back by the rejection count so the KV cache accounting stays honest. There is even a staleness guard here: rejections that predate a preemption rollback are skipped, because applying them twice would corrupt the token counts.

Grammar gates: structured output

The second tower is the constraint system. A request that carries structured_outputs gets routed to the StructuredOutputManager, whose grammar_init lazily instantiates the backend. The manager keeps exactly one backend per engine, a decision documented in a comment about not supporting per-request backends in v1, and picks xgrammar, guidance, outlines, or lm-format-enforcer based on what the first constrained request asks for. Compilation is asynchronous by design: _create_grammar submits to a thread pool sized at half the machine’s CPUs, and the returned future carries failures back to the scheduler so a malformed schema fails only its own request rather than wedging the engine. There is even a ReDoS guard in the utility module: regex compilation runs under a timeout, because a nested quantifier in an untrusted schema can explode the DFA state space and hang a worker.

The interesting part is what happens once the grammar exists. Each step, the scheduler asks for a bitmask through grammar_bitmask, a tensor with one row per request plus one row per speculative position, because a grammar must constrain every candidate the drafter proposes, not just the bonus token. Large batches above 128 requests get their rows filled by a worker pool in chunks of sixteen; smaller batches fill serially to avoid the dispatch overhead. Before any row is filled, _get_constraint_bounds decides where the grammar should start constraining, which is how reasoning models get to think freely in natural language and only meet the grammar once they open the content section; while reasoning, the mask instead unblocks the reasoning-end tokens so the model can finish its thought.

The GPU side consumes this in apply_grammar_bitmask. The scheduler sends masks compacted to only the constrained requests, in scheduler order, while the runner’s batch has its own order and per-request speculative offsets, so the function rebuilds an index mapping that walks both orderings, accounting for the spec-token offset of every request, and copies the sorted mask to the device asynchronously. The final masking is a single call to xgrammar’s apply_token_bitmask_inplace kernel, which zeroes the logits of tokens the FSM cannot accept. Then the interaction between the two towers becomes visible: speculative padding tokens are stripped before they reach any grammar, draft positions each get their own mask row, and if a drafter proposed something the grammar forbids, the scheduler’s accept path through accept_tokens feeds the surviving tokens to the per-request GrammarMatcher, advancing the FSM one token at a time and rolling back cleanly when the rejection sampler invalidated a suffix.

Telemetry and tuning

None of this machinery is worth much if operators cannot see whether it is helping, which is where SpecDecodingStats comes in. Every step, the scheduler aggregates draft counts and accepted counts, including a per-position breakdown where position zero is usually near-perfect and later positions decay, a histogram that tells you whether your draft length is too ambitious. The logging wrapper even relabels these counters for diffusion language models, which reuse the spec-decode data path with different semantics. The scheduler can also emit per-request metrics for operators who want to see which individual workloads benefit from drafting, and the dynamic spec-decode utility can schedule lookup tables that adjust the draft length with batch pressure, trading acceptance for throughput as the GPU fills.

The detail view: scheduler-side spec-token accounting and grammar bitmask generation, the drafter dispatch tree, the rejection sampling kernel chain with its recovery path, the xgrammar compilation and FSM advance loop, and the telemetry that ties acceptance rates back to operator dashboards.

What makes this slice of the codebase worth studying is the discipline of the interfaces. Every drafter, whatever its internals, exposes the same propose surface; every grammar backend, whatever its FSM, answers the same four questions: compile, fill a mask, accept tokens, roll back. The scheduler neither knows nor cares whether the draft came from a numba loop over the request’s own text or a second model’s forward pass, and the sampler does not care whether the mask came from JSON schema or a regex. That separability is the reason the project can add a new drafter or backend in a few hundred lines, and it is the pattern to steal if you are building inference systems of your own. Pair this tour with our visits to the router logic in /litellm-2/, the scheduler and memory placement logic in /ollama-3/, and the memory-layer redesign in /llama.cpp-2/, and you will have seen the same lesson four ways: in serving engines, the levers that matter live at the boundaries where scheduling meets sampling.

Enjoyed this post? Never miss out on future posts by following us