Our first walk of microsoft/onnxruntime stayed at the architectural level: the versioned C API, the fusion transformers, MLAS, the CUDA provider, and the training stack. That post ends where every session ends up on a hot path — a call to Run() with a dictionary of feeds — and leaves the most performance-critical half of the codebase unexplored: what actually happens between that call and the last kernel’s write. This tour follows exactly that half. It is a story with three distinct layers. At the top, InferenceSession::RunImpl validates options and hands the plan to an executor. In the middle, ExecuteThePlan walks the graph across a small set of device streams, checking barriers before each node fires. At the bottom, an ExecutionFrame and a best-fit-coalescing arena have already pre-computed — at session initialization time — where every byte of every intermediate tensor will live, so that a steady-state inference run performs almost no allocator decision-making at all. As always in this series, this is an educational read of published MIT-licensed source on the main branch; the first post at /onnxruntime/ covers the API surface and provider layer that this machinery serves.

The headline is that ONNX Runtime treats allocation as a compile-time problem, not a runtime one. Two planners run during Initialize, not during inference: one assigns nodes and weights to devices, the other traces every buffer’s lifetime and folds the resulting sizes into reusable memory patterns. By the time Run() is called, the expensive questions have yes-or-no answers cached in the session state, and the per-run work reduces to bookkeeping — which is why the code can afford a comparatively elegant stream walk in the middle.

Overview of the execution machinery: RunImpl enters the executor, ExecuteThePlan walks nodes through the StreamExecutionContext and DeviceStreamCollection down to device handles like CudaStream, the planners and ExecutionFrame pre-compute every allocation into the BFCArena, and KernelRegistryManager plus the Memcpy kernel and the ThreadPool round out the launch path.

Reading the overview from left to right:

• inference_session.cc is the public face: Load for file paths, RunImpl for the hot path, and RunAsync at 4141 for the callback variant.
• sequential_executor.cc holds ExecuteThePlan and ExecuteKernel — the modern stream-based execution loop that replaced the old class-based executors.
• stream_execution_context.cc defines the per-run context whose constructor starts at line 12, with DecCountDownBarrier enforcing launch order.
• device_stream_collection.cc builds one stream per EP stream id: SetDeviceStream fills the set, GetStreamForDevice looks it back up.
• cuda_stream_handle.cc is the concrete NVIDIA handle: EnqueDeferredCPUBuffer and CleanUpOnRunEnd manage host-side work on device time.
• allocation_planner.cc at 2600+ lines contains PlannerImpl::CreatePlan and the DeviceBasedPartitioner.
• ort_value_pattern_planner.cc traces lifetimes through TraceAllocation and emits GeneratePatterns.
• execution_frame.cc materializes buffers per run: AllocateAsPerAllocationPlan follows the precomputed recipe.
• bfc_arena.cc is the best-fit-coalescing allocator — Alloc, SplitChunk, Merge, and a stream-aware variant at line 850.
• kernel_registry_manager.cc resolves ops to kernels: SearchKernelRegistry and CreateKernel.
• memcpy.cc is the tiny Memcpy::Compute kernel that bridges devices.
• threadpool.cc implements ParallelForFixedBlockSizeScheduling over the Eigen-derived work-stealing pool.

Before anything runs: kernels, plans, and the fallback

Initialization is where the engine earns its steady state. After the graph transforms have fused and rewritten the model — the subject of the first tour — SessionState::FinalizeSessionState locks in three artifacts. First, kernels: SessionState::CreateKernels asks the KernelRegistryManager to construct a kernel instance per node, and that manager’s SearchKernelRegistry walks every registered registry comparing op, version, type constraints, and execution-provider capability until one matches. This is why a session cannot gain provider support mid-run: the kernel objects were built for one specific partition, with attributes and constant initializers already attached. Second, plans. PlannerImpl::CreatePlan runs the topological walk twice, essentially — once to assign every node to a device via the DeviceBasedPartitioner, deciding where weights and activations will live, and once to build the per-node allocation recipe. Where a provider cannot claim a node, GetCpuPreferredNodes keeps the CPU EP from over-reaching by preferring nodes whose weights are small enough to matter and whose placement would not trigger a cascade of copies.

Third, memory patterns. The OrtValuePatternPlanner replays the plan in miniature: every TraceAllocation and TraceFree records a buffer’s lifetime, and GeneratePatterns folds them into a MemoryPatternGroup — a block layout in which activations that never overlap can share one arena region. The session state caches these groups keyed on the shapes of the input tensors: UpdateMemoryPatternGroupCache stores the first match it computes, and ResolveMemoryPatternFlag decides whether patterns are even legal, which depends on whether the model has run-to-run shape variability. Fixed-batch deployments — the common inference server case — hit the cached pattern on run two and reuse it forever. This is the same weight-reuse philosophy we saw in the TurboMind block manager of lmdeploy-2, just expressed in C++ allocator terms instead of token-block terms.

One run, many streams

The executor itself is refreshingly small once the precomputation is done. ExecuteThePlan builds a fresh ExecutionFrame, feeds the inputs, and drives the node loop; ExecuteKernel is the per-node body that resolves OrtValues, wraps them in a kernel context, and calls Compute. What makes this loop modern is that it is stream-first: the engine no longer pretends there is one global execution thread. Instead, DeviceStreamCollection holds one stream per logical execution lane — typically one per device, plus extra ones a model may have declared with a loop or an if — and SetDeviceStream wires concrete handles into those lanes at run start. A run that touches two devices gets two lanes; a node on lane A that consumes a tensor produced on lane B must wait for a notification, and that is exactly what the StreamExecutionContext tracks with its notification table and countdown barriers.

The abstraction that keeps this device-neutral lives in stream_handles.h: a Stream with CreateNotification and Flush, and a Notification the consumer can wait on. The framework ships a default implementation in plugin_ep_stream.cc — its WaitNotificationOnDevice is an honest placeholder that busy-waits on host — and providers override it. The CUDA override is the instructive one: CudaStream wraps a cudaStream_t, and its CreateNotification produces a real event-based cross-lane wait. The subtle piece is host-side work: when a kernel needs to write a value that only becomes known mid-run (a shape, a dynamic dimension), the CUDA path cannot block the device, so EnqueDeferredCPUBuffer parks the host buffer in a queue that CleanUpOnRunEnd drains after the stream has actually finished — the DeferredCpuAllocator at line 11 exists to make that buffering transparent to the caller. Cross-lane data movement itself is just a node: the Memcpy kernel gets inserted by the planner wherever a consumer’s device differs from its producer’s, and GetStreamForDevice decides which lane executes it.

The frame and the arena: where every byte comes from

On the first run with a new input signature, the ExecutionFrame earns its keep. IExecutionFrame::Init binds feeds to OrtValue slots; AllocateMLValueTensorSelfOwnBuffer handles the naive case where a tensor owns its own buffer; and AllocateAsPerAllocationPlan handles the fast case — it consults the precomputed plan, and when a memory pattern exists, every activation in the group is carved out of a single arena reservation at fixed offsets. CreateNodeOutputMLValueImpl then hands those regions to the kernels as writable tensors, and ExecutionFrame::GeneratePatterns is the bridge that turns a planner’s group into real buffer geometry. Underneath all of it sits the arena. BFCArena — best-fit with coalescing, the same design PyTorch’s caching allocator made famous — takes big chunks from the device allocator via Extend, splits them through SplitChunk into size-ordered bins, and rejoins freed neighbors through Merge. The payoff shows up on the second run: Free returns blocks to bins instead of the device, so steady-state inference does zero device allocator calls for activations. The stream-aware variant at AllocOnStream sharpens this per lane, and when memory pressure bites, DumpMemoryLog prints the bin-by-bin accounting that turns “why is my GPU OOM” into an answer.

The last layer is concurrency inside kernels themselves. Most heavy kernels do not monoprocess their input; they chop it into blocks and call ParallelForFixedBlockSizeScheduling on the intra-op pool, which schedules fixed-size chunks across worker threads with the deterministic block ordering the profiler-facing ThreadPoolProfiler machinery records. The pool is a vendored, heavily modified Eigen work-stealing pool: ThreadPoolTempl drives lock-free queues implemented as the spsc RunQueue at line 495, with the ThreadPoolLoop shim at line 403 bridging to Eigen’s own primitives. So a GEMM on CPU is three systems deep — the MLAS kernel from the first tour, the pool feeding its blocks, and the arena holding its operands — and all three were shaped by the same decision: do the expensive thinking once, at Initialize, so Run() only executes.

From Install to Your First Model

Everything above is on by default after pip install onnxruntime or onnxruntime-gpu. The knobs that expose the machinery: session options control intra-op and inter-op thread counts, arena settings like arena_extend_strategy choose between growing the arena in kArenaAllocation steps and reusing device allocator memory, and enable_mem_pattern toggles the whole pattern system for debugging. When a run behaves oddly, ORT_DEBUG_NODE_IO and the memory-pattern dump give the two most direct windows into this layer — the arena’s DumpMemoryLog output is genuinely one of the most readable allocator reports in the ecosystem.

Honest limits: this walk focused on the CPU-plus-CUDA path because that is where the code is complete; the WebGPU, NNAPI, and Core ML providers implement the same Stream/Notification contracts with their own quirks that we did not read. We read code, not benchmarks — the “allocation is pre-computed” claim is structural, but real-world wins depend on shape stability, and a model with dynamic batch sizes will spend its runs in the slower self-owned-buffer branch. And the executor is mid-migration: the old class-based executors are gone, the stream-based loop above is the present, and the training stack shares it with extensions we deliberately skipped. For context: vLLM and SGLang solve the same memory-recycling problem at the request level with paged KV blocks, TensorRT-LLM precomputes allocation inside an engine build step, and OpenVINO answers on CPU with its own plugin allocator scheme. ONNX Runtime’s answer — plan once, pattern-cache, arena-recycle, stream-barrier — is the most explicit and most inspectable of the lot, which is exactly why it rewards this second reading.

Enjoyed this post? Never miss out on future posts by following us