Our first walk of apache/tvm followed the compiler’s happy path: Relax programs built by the BlockBuilder, rewritten by fifty-six C++ passes, lowered into TensorIR, and scheduled by dlight rules. That tour stopped, deliberately, at the boundary marked “executable” — the moment the compiler hands its output to something that runs. This visit crosses that boundary, because the runtime is where TVM’s design philosophy becomes visible in a different way: a compiler can afford megabytes of machinery, but a runtime that ships inside a model library must be small, predictable, and inspectable. TVM’s answer is a register virtual machine with exactly four opcodes, a sectioned file format, a registry of builtin services, and — the part that has grown fastest recently — a family of attention-cache objects that turn the VM into an LLM serving substrate for projects like MLC. Reading the runtime after the compiler is like watching a play after reading the script: every instruction you meet was emitted by a pass you already know, and every oddity has an author. As always, this is an educational tour of published source on the main branch, Apache 2.0 licensed.
The one-sentence summary: relax.build compiles a module into a bytecode-carrying VMExecutable, VirtualMachineImpl interprets that bytecode in a single RunLoop, and everything the interpreter cannot do inline — allocate tensors, check shapes, run attention over a paged cache, replay CUDA graphs — is a registered function in the vm.builtin.* family.
Overview of the Relax VM runtime: relax.build runs codegen and linking, the CodeGenVM emits four-opcode bytecode into a sectioned executable, the VirtualMachine loads and interprets it, builtin calls handle allocation and shape checks, and the LLM support layer drives paged KV caches and CUDA graphs.
Reading the overview from left to right:
- relax.build orchestrates the build through _vmcodegen at line 53 and _vmlink at line 99.
- CodeGenVM turns every Relax expression into instructions, with the call path at line 138.
- LowerRuntimeBuiltin rewrites memory allocations into VM calls before codegen sees them.
- the Opcode enum defines the whole instruction set: Call, Ret, Goto, If.
- the VMExecutable serializes itself section by section and reloads through LoadFromBytes at line 209.
- VirtualMachineImpl runs frames through InvokeBytecode at line 629 and RunLoop at line 779.
- the builtin registry starts with alloc_storage and grows through match_shape at line 134.
- KV-state builtins expose attention cache operations to compiled models.
- PagedAttentionKVCache manages page pools with prefill and decode function tables at line 330.
- the CUDA graph builtin captures whole VM closures into replayable GPU graphs.
From module to bytecode: the build path
relax.build is the front door, and its three stages are worth naming. Stage one, _vmcodegen, runs the compiler’s normal pass sequence and then applies CodeGenVM, a visitor over Relax expressions whose case analysis doubles as the definition of the instruction set. A function call becomes an EmitCall with the callee as a function index; the conditional at line 186 becomes a call to the builtin read_if_cond followed by an EmitIf with jump offsets; and memory allocation is rewritten first by LowerRuntimeBuiltin, which swaps Relax’s memory.alloc_storage for the VM’s own op so codegen can emit alloc_storage at line 362. Stage two, _vmlink, fuses the bytecode module with the compiled kernel library into one deployable artifact. The instruction format itself is a study in economy: the Opcode enum has four members, and Instruction is a tagged union whose arguments pack registers, immediates, constant indices, and function indices into fixed-width words — the ArgKind scheme at line 92 with bit-level limits spelled out as constants. The result is an executable you can actually read: AsText and AsPython print disassemblies, and the save path SaveToBytes writes global, memory-scope, constant, and code sections in order, so the artifact is a plain binary you can diff across builds. Persistence is symmetric: WriteToFile and LoadFromFile wrap the byte stream with a magic header, and the Python disassembly at AsPython renders every function with its register-level instructions, which makes the bytecode an excellent teaching artifact — students can compile a two-layer network and print it. Two build-time helpers round out the story: _auto_attach_system_lib_prefix renames symbols so multiple executables can coexist as system libraries, and _filter_tir sorts out which kernels belong in the linked library, a small janitor that keeps the boundary between VM bytecode and device code clean.
The interpreter: one loop, four opcodes
The runtime half lives in VirtualMachineImpl. Calling a compiled function walks a short path: the Python VirtualMachine wraps the runtime module, its getitem at line 102 resolves a function name, and arguments are converted and dispatched into InvokeBytecode, which pushes a VMFrame and enters RunLoop. The loop is a switch over the four opcodes — a dispatch that would look austere next to any Python interpreter, and that austerity is the point: the whole execution model fits on one screen. Calls to packed functions route through RunInstrCall, which invokes either a VM function from the function pool or an external PackedFunc, including Python functions registered back into the runtime via register_py_func. The builtin registry is the interpreter’s standard library: alloc_storage claims raw bytes on a device, match_shape asserts or records dynamic shapes so symbolic dimensions become runtime checks, tuple plumbing comes from tuple_getitem at line 707, and initialize_default at line 47 wires parameter-loading modules at startup. Control flow beyond the four opcodes is delegated rather than duplicated: shape-dependent branching consults the shape heap allocated at line 80, closures travel as ordinary values through invoke_closure and its stateful sibling, and arity queries like _GetFunctionArity at line 914 let Python-side callers introspect compiled functions before binding arguments. The Python side adds conveniences the C++ should not know about — set_input and invoke_stateful separate argument binding from execution so CUDA graphs can replay, save_function at line 123 freezes a bound callable, and time_evaluator at line 344 benchmarks with warmups. The test file reads as executable documentation of all of it, starting from test_vm_compile_simple and reaching recursion and closures by test_recursion at line 801.
The LLM layer: caches, attention, and graphs
The newest and most consequential part of the tree is the LLM support layer, which turns the VM from an inference-shell into serving infrastructure. The abstraction hierarchy starts at KVStateObj: AttentionKVCacheObj declares what a compiled transformer may ask of its cache — append tokens, run attention, checkpoint and restore sequences — and RNNStateObj at line 382 covers the stateful non-attention case. The registrations in kv_state.cc read like a transformer’s forward pass offered as builtins: fused-QKV attention, self-attention, cross-attention, shared-key attention, MLA-style appends, and in-place output merges. The flagship implementation is PagedAttentionKVCache, whose internals will look familiar to readers of vLLM’s paged design: a pool of physical pages with free_page_ids at line 199, per-sequence block lists, sliding-window accounting, tree-mask prefill for speculative decoding, and a function table of prefill and decode kernels at line 330 supplied by the attention backend picker, which currently selects flashinfer and wraps its plan/run split. The cache also thinks beyond a single process: sequences can be exported and imported page group by page group for checkpointing, GetNumAvailablePages gives the scheduler its admission signal, and the disaggregation hooks prepare receive-side state for KV transfer across machines, with an NVSHMEM-based page-to-page transfer as one transport. On top sits GPU scheduling: the CUDA graph builtin keys captures by function index and symbolic shape variables through CUDAGraphCaptureKey at line 40, runs the closure once under stream capture, and thereafter replays the recorded graph whenever shapes match — the same trick DeepSpeed’s inference engine and every high-throughput server now consider mandatory, implemented here as a dozen lines of VM extension. Parameter handling closes the loop: param.py saves and reloads parameter dictionaries so artifacts stay weightless until deployed.
From Install to Your First Model
Build TVM from source with LLVM enabled — the first tour’s build instructions still apply — and the whole runtime is then three lines: Enjoyed this post? Never miss out on future posts by following us ex = relax.build(mod, target="llvm"), vm = relax.VirtualMachine(ex, tvm.cpu()), out = vm["main"](data). Be honest about the trade-offs you inherit. The VM is a bytecode interpreter, not a JIT: dynamic control flow costs a dispatch per instruction, which is precisely why hot compute is pushed into compiled kernels and why the graph builder’s dataflow blocks matter — they tell the VM what can stay in registers and what must spill to storage. CUDA graph replay requires the stateful input protocol rather than direct calls, the paged cache currently expects its attention functions from flashinfer on NVIDIA hardware, and the executable format is versioned but not a cross-version guarantee. What you gain is a runtime you can read end to end in an afternoon, an artifact format you can disassemble, and a serving substrate whose memory planner, cache, and scheduler are all visible seams rather than hidden magic. Set it beside the bytecode-free execution plans of ONNX Runtime, the AOT-compiled closures of ExecuTorch, and the kernel-first philosophy of Triton, and the trade space of inference runtimes comes into focus: interpret, precompile, or lower everything — TVM ships all three in one tree and lets you watch each one work. 
