Every framework we have toured in this series — Triton for kernels, vLLM for throughput, MLC for cross-platform compilation — assumes you have an operating system, a Python runtime, and memory to spare. ExecuTorch, PyTorch’s edge sibling, drops all three assumptions: its target machines are microcontrollers, DSPs, and phone NPUs where your whole model must fit into a memory budget you declare in advance. The price of that constraint shapes every file in the repository, and it produces one of the cleanest ideas in inference engineering: the allocation plan is computed at export time, on a machine that has plenty of RAM, and shipped inside the artifact itself. The runtime never allocates a tensor at inference — it replays a script. This tour follows a .pte file end to end: the exir export flow that partitions, plans, and serializes it; the loader that memory-maps it; the Method that executes it one step at a time; the backend contract that hands subgraphs to vendor libraries; and the LLM runner that turns all of it into a chat loop. As always, this is an educational read of published source on the main branch, under the BSD-style license.

What makes the design worth studying is how much it refuses to do at runtime. There is no scheduler, no batcher, no memory arena hunting for blocks mid-inference — all of that was decided by a Python pass with the whole graph in view. The C++ side stays small enough to read in an afternoon, which is exactly the point for software that ships inside hearing aids and drones.

Overview of the .pte path: exir partitioners carve subgraphs to backends, the memory-planning pass assigns every tensor a spec, serialize_for_executorch writes the flatbuffer, Program loads it, Method replays the plan through the BackendInterface, and the Module plus LLM runner wrap it into an application.

Reading the overview from left to right:

• partitioner.py defines the abstract Partitioner whose call at line 66 runs partition at 75 and returns a PartitionResult with lowered subgraphs.
• memory_planning_pass.py is where allocation becomes data: MemoryPlanningPass runs at line 280 with a multi-method variant at 346.
• memory_planning.py computes lifetimes with update_all_tensors_lifetime and validates them through verify_storage_reuse at 165.
• _serialize.py is the exit door: serialize_for_executorch writes the flatbuffer program and its external data segments.
• program.cpp is the loader: Program::load verifies the file, num_methods at 353 counts entry points, and load_method at 384 builds a Method.
• method_meta.cpp reports what a method needs — input shapes, output shapes, memory plan sizes.
• method.cpp is the engine: Method::init wires kernels and delegates once, step at 1723 and execute at 1780 replay the plan.
• interface.h is the vendor contract: BackendInterface with init at 84 and execute at 99.
• module.cpp is the friendly C++ API: Module::load picks a DataLoader strategy, load_method at 543 prepares an entry point.
• text_llm_runner.cpp is a complete chat loop: generate drives prefill at 284 and per-token decode.
• text_decoder_runner.cpp runs one method step per token through the sampler.
• memory_allocator.h and hierarchical_allocator.h are the bump allocators that hold every planned buffer.

Export: edge dialect, delegates, and a plan for every byte

The Python half of the repo answers a question most runtimes answer with heuristics at load time: how much memory does this model need, and where does each tensor sit? The flow starts with torch.export and a descent into the edge dialect — operator decompositions that rewrite ATen ops into the edge subset with explicit dtype and layout. Then partitioning: a backend implements the Partitioner interface, declares which ops it supports, and the exporter slices matching subgraphs out of the graph, wrapping each in a lowered module that the backend compiles however it likes. Whatever the backend cannot claim stays as a portable-kernel subgraph, so a Core ML or QNN delegate can sit beside reference C++ kernels in the same method.

The distinctive part is the planning pass. MemoryPlanningPass walks the traced graph, and for every node that materializes a tensor it attaches an allocation spec at line 229 — size, dtype, alignment, and a lifetime computed from when the tensor is last read. The core machinery in memory_planning.py collects specs from nodes, then solves a two-dimensional packing problem: tensors whose lifetimes do not overlap can share one buffer, and update_all_tensors_lifetime at line 548 supplies the intervals the packer needs. The result is a set of named memory plans — each a fixed-size pool with fixed offsets — and a verifier at line 165 that refuses to emit a program whose planned buffers overlap in invalid ways. Serialization then writes the whole story into the .pte: the flatbuffer program from serialize_for_executorch at line 44 plus constant segments laid out so a device can consume them directly. It is the same precompute-everything philosophy we saw in ONNX Runtime’s planners in the onnxruntime-2 tour, taken to its logical endpoint: not cached per input signature, but baked into the file.

Loading: segments, verification, and zero-copy

The C++ half begins with Program::load, and its first act is paranoia. The loader checks the flatbuffer identifier, verifies the whole buffer with a flatbuffers verifier, parses the extended header, and only then hands back a Program. The file is organized as segments — program, constants, mutable data — described by segment info records, and the loader never copies them: it keeps a pointer into whatever the DataLoader provided, which can be a file read, an mmap, or memory embedded at link time. PteDataMap indexes the named data so constants land where the plan expects. A Program can hold several methods — the LLM use case compiles prefill and decode as separate entry points — enumerated by num_methods at line 353 and get_method_name at 364, each loaded independently through load_method at 384, which consults method_meta for the shapes and plan sizes it must reserve.

Execution: one method, one step, one delegate call

Method::init at line 937 is where the plan meets real memory. The caller supplies a hierarchical allocator — a stack of named pools sized exactly as the export promised — and init carves every planned buffer out of those pools without a single dynamic allocation. It then walks the graph’s instructions: portable kernels are resolved by name against the registered kernel library, and each delegated subgraph is initialized through the backend’s init at line 84, which compiles the subgraph into an opaque DelegateHandle. From there, execution is a replay: Method::step at line 1723 copies inputs into their planned slots, execute at 1780 walks the instruction list — kernel calls through the EValue variant, or execute at line 99 on the backend for delegated spans — and every write lands in memory whose address was decided in Python. The TensorParser is the quiet workhorse mapping feed tensors onto the planned buffers, and the optional EventTracer records per-op spans when profiling is compiled in. The Module layer above smooths the ergonomics — its constructors at line 125 choose between FileDataLoader and mmap variants depending on the platform — and BundledModule goes one step further by linking the .pte into the application binary itself, which is how a watch face model ships with no filesystem at all.

Two supporting pieces make this machinery survivable on real hardware. The first is the DataLoader abstraction: because the loader only ever hands out borrowed views — the FreeableBuffer type wraps a pointer plus a destructor — the same Method code runs against a heap read on a laptop and against flash-mapped memory on an RTOS without caring which. The Module constructors pick the strategy per platform, trying mmap first and falling back to file reads where it is unavailable. The second is the Result type, which replaces exceptions with a value-or-error pattern throughout the runtime: every loader call, init call, and step returns one, and the absence of exception machinery keeps the binary small enough for the MCUs this code targets. Neither piece is glamorous, but together they are why the executor compiles with almost no standard library dependency — an explicit design goal you can feel in every header.

From Install to Your First Model

The LLM path ties the pieces together. TextLLMRunner owns two methods from one .pte — prefill and decode — and generate at line 74 runs prefill at 284 in chunks sized to the device, then loops TextDecoderRunner::step at line 40, which executes the decode method once, samples, appends the token, and invokes the callback. Nothing here batches across users — the whole design targets one conversation on one device, which is why the runner can stay a few hundred lines.

After pip install executorch, the knobs worth knowing are the memory ones: memory-planning modes choose between contiguous pools and per-plan pools, --memory-planning output can be inspected as a text layout, and backends like XNNPACK or Core ML register themselves as partitioners you add to the export call. Honest limits: we read the loader and method hot path, not the kernel libraries or the vendor delegates — the QNN and Core ML trees are large projects of their own. We benchmarked nothing; the zero-copy claims are structural, and real devices add alignment surprises that the dim-order passes only partially hide. And the executor is single-threaded by design — parallelism exists only inside backends. Compared with OpenVINO and ONNX Runtime, which carry their allocators and schedulers into the runtime, ExecuTorch’s answer is radical precomputation: the .pte is not just a model format, it is a memory map — and that is why the smallest devices in this series run it.

Enjoyed this post? Never miss out on future posts by following us