Our recent tours have lived close to the GPU: TensorRT-LLM’s process lattice, vLLM’s core engines, SGLang’s router fan-out. This visit goes somewhere quieter and, arguably, harder — the CPU. OpenVINO has been shipping since before “inference framework” was a job title, and its CPU plugin is one of the most complete hand-optimized graph executors anywhere in open source. The question this tour asks is the user’s-eye question: what actually happens between core.compile_model(model, "CPU") and request.infer()? The answer runs through a facade that hides its own implementation, a plugin registry loaded from an XML file, a content-addressed cache that can skip compilation entirely, a graph engine with a strict ordering of setup passes, a factory of node classes that now includes attention and MLP kernels, and a threading layer that pins stream threads to physical cores. As always, this is an educational walk through the published code of openvinotoolkit/openvino on the master branch, under the Apache-2.0 license.

The headline discovery is how literal the word “compile” is here. compile_model is a genuine compiler invocation — clone the graph, run transformation lists, pick primitive descriptors per node, resolve in-place edges — and the result is hashed so a second process can import a serialized blob instead of repeating the work. The second discovery is that one CompiledModel owns not one graph but one per stream, each fully configured and ready to execute, with a guard object handing them out and re-initializing on shape changes.

The overview: the ov::Core facade delegates to CoreImpl, which hashes models into the ModelCache and dispatches to the CPU plugin; the plugin builds a CompiledModel that owns one Graph per stream, wraps each SyncInferRequest in an AsyncInferRequest backed by the generic stage machinery, and executes everything on CPUStreamsExecutor threads.

Reading the overview from left to right:

• The public facade at src/inference/src/cpp/core.cpp — an inner Impl class that extends CoreImpl, keeping the header clean while all logic lives in the dev layer.
• CoreImpl owns the plugin registry, the device dispatch table, and the compile/import entry points.
• ModelCache computes a model hash and stores or imports compiled blobs on disk.
• The CPU plugin’s Plugin::compile_model clones, transforms, counts streams, and hands a CompiledModel back.
• CompiledModel builds one Graph per stream and manufactures requests through create_infer_request.
• The Graph configures, allocates, and executes a topological node sequence drawn from the NodesFactory registry.

A facade, a registry, and a content-addressed cache

Start where every user starts. Core’s constructor allocates an inner Impl — and that Impl is not a pimple of raw fields but a subclass: Core::Impl : CoreImpl at line 60. Everything the public class appears to do, it actually inherits from the dev-layer implementation in core_impl.cpp. The first thing CoreImpl does is read a plugin XML: register_plugins_in_registry parses device names, library paths, and device properties into a registry, and build_dispatch_entries turns the wildcard patterns — CPU, GPU, HETERO:GPU,CPU — into an ordered dispatch table, with get_internal_plugin_name resolving names like BATCH to their real plugin. Asking for a device that maps to another device is resolved here, long before any model is seen.

read_model is thin — it routes through the model reader at src/inference/src/model_reader.cpp, which sniffs XML versus ONNX versus a pre-serialized blob and deserializes into an ov::Model. The weight is on compile_model: four overloads taking a model, a device string, and configuration as remote context, path, or plain map, funneled into one CoreImpl method. Before the plugin is ever called, apply_auto_batching rewrites the device — CPU with the AUTO_BATCH hint becomes the BATCH plugin wrapping CPU — and the config is normalized. Then compile_model_and_cache consults the cache: get_cache_model_path builds the directory, get_model_hash_sync_key builds the sync file name, and the actual fingerprint is ModelCache::compute_hash — a digest over the model itself plus every relevant compile option. On a miss, the plugin compiles and the result is stored behind a CompiledBlobHeader that versions the format; on a hit, import_model hands the bytes to the plugin’s import path and the entire compilation is skipped. For a large LLM graph this is the difference between minutes and milliseconds on process start.

compile_model: clone, transform, count your streams

Down in the plugin, Plugin::compile_model opens with something that surprises people: the model is cloned before a single transformation runs. The reason is API honesty — the user’s ov::Model must remain exactly as they built it, because they may compile it again for another device or read its ops for debugging. The clone then walks a fixed transformation sequence: UpToLpt lifts low-precision conversions upward so quantized data flows as int8 as far as possible, PostLpt cleans up what remains, the stream count is decided by calculate_streams from the performance hint and core topology, Snippets fuses eligible subgraphs into JIT-generated kernels, and CpuSpecificOpSet validates that every operation has a CPU implementation. Only then does the CompiledModel get constructed. The plugin also exposes get_property and query_model, the latter letting tools ask which nodes would fall back before paying for compilation.

The CompiledModel constructor is where the concurrency model appears. It first creates its task and callback executors — the task executor is a stream executor sized by the stream count, and every asynchronous request produced by this model shares it. Then comes the most distinctive choice: one graph per stream, each fully built and configured, so two in-flight requests never fight over mutable primitive state; sub-stream handling covers the case where a model is internally split across streams. Access is arbitrated by get_graph, which returns a GraphGuard::Lock — an RAII handle — and picks the graph by socket affinity so a request lands on the graph whose memory sits nearest. On a shape change the guard re-runs Init and Activate instead of rebuilding the world. Request creation is a pair of factory methods — create_sync_infer_request and the async wrapper at line 239.

Configure: making nodes executable

The graph’s own lifecycle lives in graph.cpp, and its heart is Configure — an ordered list of passes where the order is the whole idea. SortTopologically fixes an execution sequence first, because everything downstream assumes it. InitNodes replaces the framework-agnostic ops with plugin node objects, resolved through the registry: NodesFactory maps operation type names to constructors via the INTEL_CPU_NODE macro, and the include list at the top of the file is a small museum — alongside Convolution and MatMul sit the LLM-era registrations pulled in at lines 119 through 127: RoPE, scaled attention, RMS norm, QKV projection, paged attention, and an MLP node.

Then the memory-planning passes. ApplyCommonGraphOptimizations folds and fuses what it can; InitDescriptors gives each node a chance to declare candidate implementations, from which InitOptimalPrimitiveDescriptors selects — the decision procedure is selectOptimalPrimitiveDescriptor in the node base class, weighing precision support against conversion costs. ResolveInplaceDirections and the node-side canBeInPlace decide which outputs can be written over their inputs, saving a copy per eligible edge — a meaningful win when tensors are activations of a transformer layer. ResolveEdgeConflicts inserts copies where an edge serves consumers wanting different memory layouts, ShareReorders deduplicates identical layout conversions, and ApplyImplSpecificGraphOptimizations lets nodes do last-mile surgery. Only after Configure does Activate allocate the actual memory and create the executable constants. Nothing moves until every node has a primitive it can run.

The detail view: the API tier of facade and model reader; the runtime tier of CoreImpl, ModelCache, the compiled-model and request base interfaces, and the generic async stage machinery; the CPU plugin tier of plugin, CompiledModel, and its sync/async requests; the graph engine of Graph, NodesFactory, and the node base class; and the threading tier of the CPU stream executor.

Walking the detail diagram, a request crosses every tier in order. The user’s tensors enter through the base interfaces — ISyncInferRequest handles port lookup and tensor plumbing, while ICompiledModel holds the executor pairs every request shares and its create_infer_request is the template the plugin overrides. Above the sync request sits the generic async wrapper: its constructor builds a two-stage list — run the sync infer on the task executor, then fire the user callback on the callback executor — and start_async just runs the first stage; make_next_stage_task chains to the next stage, and the final task settles the promise and invokes the callback. The CPU plugin subclasses this once in async_infer_request.cpp to add a fast path: when the model runs on a single stream, infer skips the generic stage list and executes directly on the stream executor — one less context hop per call.

Infer: one thread per stream, pinned to silicon

The synchronous request is where tensors become work. SyncInferRequest::infer reads like the diagram: push every input into the graph via PushInputData, call Graph::Infer, and pull outputs with PullOutputData. When the model was compiled for multiple streams, the request holds sub-requests — one per stream — and sub_streams_infer splits the batch across them, each on its own graph. Inside the graph, Infer picks static or dynamic: InferStatic walks the precomputed topological order — a tight loop over the node sequence, execute per node — while InferDynamic re-checks shapes and can fall back through ReadyDynamic checks when a node cannot keep up with a new shape.

The threading layer underneath is CPUStreamsExecutor. Each stream owns a TBB arena, and the executor’s bookkeeping — a stream-id queue handed out and returned by get_stream_id — is what lets CompiledModel say “graph number three belongs to stream number three.” On scheduler entry, an observer pins the thread to a vacant core chosen with NUMA and socket awareness, so a stream that starts life next to its graph’s memory stays there; nodes can even request relocation through toNumaNode. The execute and run methods are deliberately boring — submit a task to the arena, wait — because all the sophistication lives in the pinning and the arena constraints.

From Install to Your First Model

The usual two lines — pip install openvino, then ov.Core(), read_model, compile_model, infer — now read as the whole diagram. The knob this tour makes legible is INFERENCE_NUM_THREADS and the performance hint: THROUGHPUT sizes multiple streams, and everything downstream — graph count, request splitting, arena sizing — follows from that one decision during calculate_streams. Latency-minded users get one stream, and the async fast path keeps the dispatch overhead near zero.

Honest limits: this walk is code reading, not benchmarking — the performance claims here are structural, not measured. We toured the CPU plugin because it is the most self-contained; the same facade fronts GPU and NPU plugins whose compile paths differ substantially after the cache layer. And while the LLM nodes in the factory show exactly where transformer serving entered this codebase, running LLMs on CPU well is its own discipline of quantization and scheduling that this tour only points at. For the wider ecosystem, TensorRT-LLM is the GPU-counterpart worth reading next — same questions, answered with processes and streams of a very different kind — while LocalAI and Xinference show how frameworks like this one get embedded behind serving frontends. OpenVINO’s answer — one graph per stream, configured once, executed on pinned threads — is the oldest pattern here, and still one of the cleanest.

Enjoyed this post? Never miss out on future posts by following us