Most libraries that talk to language models treat the model as an oracle: you send a prompt, you get text back, and you pray it parses. Guidance — the open-source project maintained under the guidance-ai organization and originally developed at Microsoft — takes the opposite stance. Its README calls it “an efficient programming paradigm for steering language models”: you write ordinary Python, interleave prompts and generation with system(), user(), and assistant() blocks, and constrain the model’s output with regular expressions and context-free grammars. Output that must match a schema matches that schema by construction, not by hope.

What makes Guidance worth a source tour rather than just an API summary is that the constraint machinery is not a thin wrapper. When you write gen(name="expr", regex=r"\d+([+*]\d+)*"), that string travels through a grammar-builder layer, gets serialized into an AST of node objects, is interpreted node by node against an abstract Interpreter protocol, and finally lands in an Engine that constrains logits at the token level — fast-forwarding tokens the grammar already determines.

This post walks that whole path in the current code (version 0.3.2 on the main branch). We generated two clickable architecture diagrams directly from the repository: an overview with the twelve components that matter, and a detail diagram with thirty components and forty-four relationships covering the library functions, the parsing layer, the engine internals, and the trace-and-widget subsystem. Every class and line number below was verified against the actual tree.

Overview of the guidance repository: from the package surface on the left, through grammar builders and the node AST, to the Model object and the engine-backed backends on the right.

Reading the overview from left to right:

• The package entry point guidance/__init__.py, L23 does something unusual: it swaps the module’s own class so that import guidance gives you a callable module — calling it applies the decorator.
• That decorator lives in _guidance.py, L17 as def guidance(...), with GuidanceFunction at L44 binding the Model argument into wrapped functions.
• The everyday surface — gen() at L13, select(), capture(), and the role context managers starting at role() L8 in _role.py — is aggregated by the library package.
• Grammar builders in _grammar.py, L27 produce the node tree defined in _ast.py at GrammarNode L208.
• A byte-level parser, ByteParser at L197, validates strings against those grammars offline — no API calls needed.
• Everything flows through the immutable Model class at L58, whose __add__ at L100 copies state on every lm += ....
• Model copies drive an Interpreter at L32, which hands nodes to the token-level Engine at L38.
• Concrete backends — Transformers, LlamaCpp, OpenAI, Mock — are exported from models/__init__.py, L10.

The Grammar Layer: Everything Is a Node

The foundation is guidance/_grammar.py, a module of pure builder functions: string() at L19, regex() at L23, gen() at L27, select() at L61, repeat() at L117, token_limit() at L141, with_temperature() at L156, capture() at L175, and subgrammar() at L182 for embedding a separately compiled grammar. None of these touch a model. They return node objects.

Those nodes live in guidance/_ast.py, and the hierarchy is worth reading top to bottom. ASTNode at L150 is the abstract base; GrammarNode at L208 adds the operations that let nodes compose with + and % arithmetic. The leaves are LiteralNode at L317, RegexNode at L361, SelectNode at L369 for choices, JoinNode at L397 for concatenation, RepeatNode at L420, and RuleNode at L464 for named rules. Two special subclasses — SubgrammarNode at L544 and JsonNode at L568 — carry precompiled grammars and JSON Schema definitions respectively. The same file also defines Function at L99, the wrapper that makes a @guidance-decorated function composable inside expressions, plus the RoleStart and RoleEnd chat-turn markers.

Compared with other structured-output tools: Outlines builds finite-state machines from regexes, Instructor validates after the fact with Pydantic retries, while Guidance represents the constraint itself as a first-class serializable grammar the execution engine can reason about byte by byte. The guidance/_parser.py module proves the point: TokenParser at L27 and ByteParser at L197 let you call grammar.match("expr=12+7*3") and get a parse result back with no model involved at all — a useful debugging loop for gnarly regexes.

The Model Object: Immutability by Copying

guidance/models/_base/_model.py defines the object every backend inherits from. Model at L58 holds an interpreter, sampling parameters, and a set of trace nodes; the constructor lazily requests a renderer from the registry when echo=True, so the Jupyter widget can attach early.

The heart is __add__ at L100, and its first line is the whole philosophy: self = self.copy(). Every lm += ... returns a new Model. The README is explicit about this — “Model objects are immutable, so this is a copy” — which lets you fork a conversation and run several continuations without losing the originals. After copying, __add__ applies any active role blocks and then dispatches: a plain str is parsed into nodes, a Function is invoked with the model as argument, and an ASTNode goes to _apply_node.

_apply_node at L115 does two things per node: it records trace input — RoleStart, RoleEnd, LiteralNode, image and audio inputs each get their own attribute — then it iterates self._interpreter.run(node, ...) and records each output attribute, pushing a model-state copy onto context-local event queues as it goes. That push is what makes streaming and the live widget possible without special-casing in user code.

Reading results back is deliberately boring: __getitem__ at L223 returns whatever gen(name=...) or capture() stored, __str__ at L212 renders the full transcript including the model’s response, and stream() at L157 hands you a ModelStream at L330 that delays all execution until you iterate it.

The Engine Layer: Constrained Decoding

The Model never talks to weights or APIs itself. Underneath sits the engine subsystem: Engine at L38 is the abstract engine owning a tokenizer pair, EngineInterpreter at L16 translates each GrammarNode into engine calls, EngineState at L7 tracks which bytes have been committed so parsing resumes correctly across state copies, and Tokenizer at L26 keeps the byte-to-token mapping symmetric.

This is where token fast-forwarding happens. Because the grammar often forces the next bytes — after <h1> there is nothing legal but </ before the matching tag name — the engine can consult the parser, recognize that the model has no freedom left, and insert the forced tokens directly. The README quantifies the benefit qualitatively: this “saves forward passes through the model, and hence reduces GPU usage.” The same machinery explains why the gen() docstring warns that max_tokens is “not exact when regular expression pattern constraints are present”: the engine ends generation as soon as the constraint allows.

For anyone extending the library: Interpreter at L32 is generic over its state type (Generic[S]), and raises UnsupportedNodeError at L107 when a backend cannot honor a node kind — so a remote API backend gracefully rejects grammar features it cannot enforce.

From Install to Your First Model

Everything above is one pip install guidance away — if you already have the backend for your model, there is nothing else to install. The README’s canonical first run uses a local Transformers model — for other local options see our llama.cpp walkthrough:

from guidance import system, user, assistant, gen
from guidance.models import Transformers

phi_lm = Transformers("microsoft/Phi-4-mini-instruct")
lm = phi_lm  # Model objects are immutable, so this is a copy

with system():
    lm += "You are a helpful assistant"

with user():
    lm += "Hello. What is your name?"

with assistant():
    lm += gen(max_tokens=20)

print(lm)

The roles come from guidance/library/_role.py — system() at L16, user() at L32, assistant() at L48 — each a context manager that pushes opener and closer blocks built on the Block class at L8, all rendered through whatever chat template the model uses.

Constrained generation is the same shape with one extra argument. To force a number: lm += gen("lm_age", regex=r"\d+", temperature=0.8). To force a multiple-choice answer: lm += select(["A", "B", "C", "D"], name="model_selection"). To force schema-valid JSON, pass a Pydantic model to the json() function at L13:

from guidance import json as gen_json

with assistant():
    lm += gen_json(name="bp", schema=BloodPressure)

loaded = json.loads(lm["bp"])

Repetition helpers like one_or_more() at L22 and zero_or_more() at L27 let you build list- and document-shaped grammars, and the README’s HTML-page example composes stateless @guidance functions into a full context-free grammar — head, body, headings, paragraphs — where closing tags are fast-forwarded automatically.

Backends, Testing, and the Jupyter Widget

The backend set exported at models/__init__.py, L10 covers the practical spectrum. Transformers at L581 wraps TransformersEngine at L345 for local Hugging Face models (more in our Transformers guide); LlamaCpp at L239 drives GGUF models through LlamaCppEngine at L78; OpenAI at L34 pairs the API client with OpenAIInterpreter at L15, which maps grammar rules onto the endpoint’s structured-output features; and two factory functions — create_azure_openai_model at L76 and create_azure_aifoundry_model at L220 — handle Azure deployments.

For development, Mock at L165 is the sleeper hit. Backed by MockEngine at L66 and a ByteTrie at L192, it replays canned bytes through the full grammar machinery — so Mock(b"<s>expr=12+7*3") followed by lm += grammar prints 12+7*3 from lm["expr"] with zero network calls. Combine that with grammar.match() and constraint logic becomes unit-testable in CI like any other pure function.

Finally, the echo path is a small real-time system of its own. The registry in guidance/registry/_registry.py hands out context-scoped singletons: get_trace_handler at L61 and get_renderer at L71. Every input and output attribute is recorded into a TraceNode at L242 tree managed by TraceHandler at L332, published as messages over TopicExchange at L16, and consumed by renderers — the base Renderer at L68, the rich JupyterWidgetRenderer at L276, and AutoRenderer at L586.

The detail diagram: seven subsystems, thirty components, forty-four verified relationships.

Honest Limits

The README is refreshingly restrained; so are we:

• Full grammar support is backend-dependent. The README states the constraint system “can ensure that the output conforms to any context free grammar (so long as the backend LLM has full support for Guidance)”. Remote API endpoints enforce only what their structured-output features allow, and node kinds a backend cannot honor surface as UnsupportedNodeError.
• max_tokens is approximate under regex constraints. The gen() docstring says this outright: the limit “is not exact when regular expression pattern constraints are present,” though the engine attempts to end generation as soon as the constraint is satisfied.
• Regex matches may overshoot. For variable-length expressions the model “is free to continue the expression after a complete match,” terminating only when something off-pattern is generated — the docstring notes the ending behavior may change as the project matures.
• Some integrations are explicitly unmaintained. The repository keeps models/broken_models/ for Anthropic, Cohere, Gemini, LiteLLM, Together, Vertex AI and friends — present in the tree, but not part of the supported __all__ export list.
• Fast-forwarding benefits are qualitative here. The README illustrates token fast-forwarding with local-model widget examples; we did not run latency or GPU benchmarks for this post and report none.

Enjoyed this post? Never miss out on future posts by following us