Most inference projects we have toured pick one strategy and sharpen it: vLLM optimizes GPU serving, llama.cpp squeezes CPUs, KTransformers splits work across both. IREE (Intermediate Representation Execution Environment) takes the hardest and most general path: a full compiler stack, built on MLIR, that lowers a machine-learning program through four increasingly machine-specific dialects and then executes it on CPUs, AMD GPUs via HIP, NVIDIA GPUs via CUDA, Vulkan, Metal, and even microcontrollers — from one source tree. If MIGraphX is a compiler with one vendor target, IREE is the same idea stretched across an entire industry, and its repository is the best public classroom for what “ML compiler” really means.

The key to reading IREE is a single boundary. Everything above it is compile time: MLIR code that transforms other code. Everything below it is runtime: C code with no MLIR dependency at all that loads compiled artifacts and runs them. The seam is the Hardware Abstraction Layer (HAL) — a dialect on the compiler side, a driver model on the runtime side. Once you see that, the four-dialect pyramid stops being alphabet soup: Flow shapes the work, Stream schedules it in time, HAL names the machine, VM carries the host program.

The overview below follows that pyramid down and then crosses the seam into the runtime, where a driver registry hands out devices and a bytecode VM executes the compiled module.

Overview of IREE: the Flow, Stream, HAL, and VM dialects form the compile-time pyramid; compiler stages orchestrate the transformations; at runtime, a driver registry instantiates HIP or CPU devices, executable caches hold compiled binaries, and the bytecode VM runs the serialized host program.

Reading the overview from left to right:

• The Flow dialect (FlowOps.cpp, L1256) partitions tensor work into dispatch regions; the Stream dialect (StreamOps.cpp, L3479) then wraps them into async execution regions with explicit fences.
• The HAL dialect (HALOps.cpp, L1683) names devices, command buffers, and executables, while host-side control logic lowers to the VM dialect (VMOps.cpp, L148).
• The compile stages are orchestrated top-down from a single compiler entry point and materialized by the dialect-level pass builders — the HAL transforms (Passes.cpp, L310) resolve target backends through TargetRegistry.cpp, L100.
• Across the seam, driver_registry.c, L89 registers driver factories — the HIP driver (hip_driver.c, L114) among them — and executable_cache.c, L76 prepares compiled code per device.
• The bytecode module (module.c, L892) loads the serialized VM program, and iree-run-module-main.c, L92 is the reference CLI that exercises the whole chain.

The Four-Dialect Pyramid

IREE’s compiler is a pyramid of MLIR dialects, each existing to answer one question the previous one left open. Flow asks “what work is there?” Its dispatch builder (FlowOps.cpp, L1256) traces tensor dependencies and carves the program into dispatchable regions — the unit of work every later stage manipulates. Verification like DispatchWorkgroupsOp::verify, L890 enforces that each region captures exactly what it needs, nothing more.

Stream asks “when does the work happen?” AsyncExecuteOp::build, L3479 wraps dispatches into async execution regions with explicit waits and signals, making concurrency a property of the IR rather than a hope. The Stream transforms (Passes.cpp, L178) then schedule, allocate, and elide redundant state — this is where IREE decides that two independent dispatches can overlap on the device.

HAL asks “what machine runs it?” ExecutableOp::build, L1683 wraps target-specific code objects, and CommandBufferDispatchOp::build, L1405 records dispatches the way a GPU command buffer would. The HAL transforms (the device-configuration stage at L245, main sequence at L310) assign devices, materialize interfaces, and hand executables to backend codegen selected via getTargetBackend, L100. Backends register themselves through mergeFrom (L30) — a plugin architecture that lets the LLVM-CPU, CUDA, HIP, Vulkan, and Metal generators live as swappable modules.

Finally the VM dialect asks “what does the host do between launches?” FuncOp::build, L148 and module verification (L110) produce a small register-based program that serializes to a flatbuffer — the .vmfb file IREE deploys. Everything else got compiled away; the VM keeps only control flow, function calls, and the glue that talks to devices.

Across the Seam: Drivers, Caches, Devices

The runtime never sees MLIR. It sees the HAL C API: drivers, devices, command buffers, executables. driver_registry.c, L89 stores statically registered factories, and device queries fan out across them (L209) so callers can enumerate “give me every HIP GPU” without knowing vendor details. Each driver is a self-contained directory under runtime/src/iree/hal/drivers/.

The CPU story has two flavors, and the contrast is instructive. The multithreaded device (task_device.c, L272) creates task queues and pools (L129) so dispatches run asynchronously across cores — this is the driver that makes CPU inference scale. The synchronous device (sync_device.c, L160) executes every command inline on the calling thread: simpler, slower, and the right choice for embedded or single-threaded hosts. On the GPU side, the HIP driver (hip_driver.c, L114) loads vendor symbols dynamically and manages streams, events, and device parameters.

Compiled code lands in the executable cache: create, L31 makes one per device, can_prepare_format, L67 asks whether a binary format is understood, and prepare_executable, L76 turns serialized compiler output into live code. This is the mechanism behind IREE’s cached-deployment story — compile once on a build machine, ship the artifact, load it on the target with zero JIT. Readers of our TVM and Executorch tours will recognize the same ahead-of-time philosophy; IREE’s version is unusual in that the host program is compiled too, not just the kernels.

The rest of the HAL runtime is the machinery every device shares through those vtables. Command buffers record dispatches, copies, and barriers into device-specific backends — the HIP driver, for instance, offers both immediate stream recording and CUDA-graph-style capture so a whole command sequence can be replayed with a single launch. Fences and semaphores give the compiled program a way to express the same wait/signal structure the Stream dialect baked into the IR, which means the runtime never has to guess ordering: it executes exactly the dependency graph the compiler decided. Allocators round it out with buffer pools sized at device creation, keeping steady-state inference free of per-step allocation churn — the same discipline our FlashInfer and KTransformers tours found inside their respective schedulers, here generalized behind one interface.

The VM: A Tiny Interpreter With a Big Job

The .vmfb artifact contains a flatbuffer archive of the VM program plus any embedded data. Bytecode module creation at L892 parses and verifies the archive header, and function lookup (L275) resolves exported names. Execution is a compact register machine: dispatch_begin, L1124 enters the interpreter loop, where each opcode is a small C handler — branch, call, or invoke a native import.

Native imports are how the VM talks to devices. The HAL module (module.c, L2139) exposes hal.* functions — buffer creation, command buffer recording, dispatch submission — that the compiled host program calls. When the interpreter hits hal.command_buffer.dispatch, it walks through the device vtable of whichever driver the session selected, which is how one .vmfb file runs unchanged on HIP, CPU, CUDA, or Vulkan. The same trick hosts our KTransformers friend SGLang’s CPU backends elsewhere; here it is the entire portability contract.

From Install to Your First Model

IREE builds with CMake and ships Python wheels (pip install iree-compiler iree-runtime on supported platforms). The quickest end-to-end tour uses the CLI tools:

# compile an MLIR/ONNX-sourced program to a .vmfb for the CPU
iree-compile model.mlir --iree-hal-target-backends=llvm-cpu -o model.vmfb

# run it on the local-task CPU driver
iree-run-module --device=local-task --module=model.vmfb \
  --function=predict --input="1x3x224x224xf32"

# same artifact, HIP driver, no recompile
iree-run-module --device=hip --module=model.vmfb --function=predict \
  --input="1x3x224x224xf32"

iree-run-module (main at L92) is deliberately boring: it builds a runtime session (session.c, L66), loads the bytecode module, parses inputs, invokes, and prints. That boringness is the point — the session API it exercises is the same one embedders use, and the demo directory in the runtime source has a fully annotated hello-world that explains every call. For tracing, the companion iree-dump-instruments and tracy tooling expose per-dispatch timings on device.

Honest Limits and Where IREE Fits

Limits first. IREE is a compiler stack, so its floor is higher than a runtime’s: integration means build systems, target flags, and understanding where your model’s ops land in the pyramid — the project’s own docs warn that frontend coverage (PyTorch via StableHLO paths, ONNX import) evolves continuously, and exotic ops can force you into custom dispatch work. Performance for large-scale LLM serving on NVIDIA GPUs is not its center of gravity; the vLLM-class stacks we covered — vLLM, SGLang, TensorRT-LLM — own that niche with far more serving machinery. And the HAL’s generality has a cost: peak single-vendor performance usually requires the vendor’s own library path, which is precisely what MIGraphX and FlashInfer optimize for their respective ecosystems.

As for fit: IREE is the right study object and increasingly the right tool when your deployment target list is long and unforgiving — mobile and edge devices, heterogeneous servers, safety-conscious embedded systems — and you want one artifact story across all of them. It is also, frankly, the best way to learn what a modern ML compiler does, because the four-dialect pyramid makes each transformation step legible. Between the compile-time IR layers and the deliberately MLIR-free runtime, IREE demonstrates the same lesson our inference overview keeps returning: the interesting engineering is almost always at the boundary where a portable abstraction meets hardware that refuses to be abstract.

Enjoyed this post? Never miss out on future posts by following us