If you have ever run a neural network on a phone, chances are the actual arithmetic was done by XNNPACK. It is the C library that sits underneath TensorFlow Lite, ExecuTorch, and the mobile builds of other runtimes, taking care of the least glamorous and most performance-critical part of inference: multiplying matrices with cache-friendly loops hand-tuned for each CPU flavor. Where engine-level projects like ONNX Runtime orchestrate graphs and devices, XNNPACK is the last mile — the layer where floats meet SIMD registers. And unlike GPU-oriented libraries such as ncnn (which also targets mobile, but with a Vulkan focus), XNNPACK is unapologetically CPU-first.

The library is famous for its brute-force thoroughness: hundreds of generated “microkernels”, one per tiling × ISA × data-type combination, dispatched at runtime to whatever your chip supports. The source is also a masterclass in systems design: above the kernels sits a declarative subgraph API, an optimizer that fuses nodes, a memory planner that schedules value lifetimes into one arena, and a weight-packing layer that reshapes constants into layouts the kernels can stream. This post walks that whole stack, from the public C API down to a generated NEON inner loop, using the current master tree as our map.

As always in this series, the code is the narrator: every symbol below links straight to the real line in the repository, and the diagrams were compiled from the actual file tree.

The XNNPACK flow at a glance: define nodes into a subgraph, build a runtime (which optimizes, plans memory, and packs weights), then invoke it so dispatch tables funnel packed data into ISA-specific microkernels.

Reading the overview from left to right:

• xnn_initialize, L47 — installs the process-wide allocator hooks and flips the global “initialized” bit that every later call checks.
• xnn_subgraph_add_nodes, L352 — the single choke point through which every xnn_define_* call appends a node to the model.
• xnn_define_convolution_2d, L1058 — the largest builder; validates shapes, padding, and grouping before recording a convolution node.
• xnn_create_runtime_v4, L570 — the heavyweight constructor: optimizes the graph, plans memory, packs weights, and wires the executable operator chain.
• xnn_plan_value_allocation_tracker, L210 — interval-schedules value lifetimes so all intermediates share one arena.
• xnn_finalize_weights_cache, L514 — turns the packed-weights cache into a fixed, reusable memory region.
• xnn_compute_gemm, L466 — the indirect-call glue that streams packed panels of weights and activations into a chosen ukernel.
• xnn_init_f32_gemm_config, L7056 — scans a table of per-CPU candidates and returns the best GEMM configuration for this chip.
• xnn_f32_gemm_minmax_ukernel_1x8__neon_dup_ld64, L21 — one of hundreds of generated microkernels: a 1×8-tile NEON GEMM inner loop.

A Subgraph, Not a Tensor Library

XNNPACK refuses to be a framework. There is no autograd, no device abstraction, no scheduler — you get two concepts: values and nodes. A value is a typed tensor with static dimensions; a node is an operator consuming and producing value IDs. Everything you build goes through xnn_define_tensor_value, L102, which registers a value and optionally hands back an external ID so the runtime can find it later, and then through builder functions like xnn_define_fully_connected, L1924 or xnn_define_batch_matrix_multiply, L620.

The builders are verbose by design — each one spends most of its lines on validation rather than work. xnn_define_convolution_2d, L1058 checks that padding is consistent, that group counts divide channel counts, and that the output dimensions are expressible at all, reusing helpers like xnn_compute_convolution_output_dimension, L46. This “fail loudly at build time” philosophy lets the hot path skip almost every check: by the time a kernel runs, the graph has been proven sane. Elementwise work gets the same treatment — xnn_define_binary, L175 and xnn_define_unary, L435 cover dozens of ops (add, multiply, relu, sigmoid, GELU, and friends) behind two functions, while xnn_define_softmax, L161 is a specialized node that will later be lowered to a fused exp / reduce-sum / divide chain.

This thin, explicit API is also why so many engines embed it. TensorFlow Lite’s delegate, ExecuTorch’s CPU backend, and mobile providers inside larger runtimes all translate their graph formats into these calls, keeping XNNPACK free of file-format or training opinions.

The Optimizer and the Memory Planner

Once the graph is complete, xnn_subgraph_optimize, L4853 runs a surprisingly long list of rewriting passes: merging adjacent reshapes, folding no-op binary ops, converting min/max pairs into clamps, simplifying negations, widening fp16 accumulators to fp32 when it changes accuracy, and even rewriting GEMMs whose right-hand side is transposed into a cheaper packing. None of this is optional polish — each pass removes work that would otherwise happen on every inference call.

Memory is planned with classic compiler techniques. populate_value_lifecycle, L37 records, for every intermediate value, the first and last node that touches it; xnn_plan_value_allocation_tracker, L210 then treats each value as an interval and packs the intervals into a single contiguous arena with minimal waste. Two tensors that never overlap in time can safely share bytes — on a phone, where 100 MB of activations is the difference between shipping and not shipping, this is not an optimization, it is a requirement. Allocation goes through the hooks installed by xnn_initialize, with defaults like xnn_allocate, L22 so an embedded target can swap in its own arena allocator.

Packing, Caching, and the Run Loop

The single most important idea in XNNPACK is that weights live in a different memory layout than on disk. GEMM kernels want fixed-size panels of rows so that each SIMD load streams contiguously; naive row-major weights would thrash the cache on every tile. So at setup time, every dense weight is repacked: the generic driver xnn_compute_packw_gemm_gio, L301 iterates over output-channel tiles, while the ISA-specific work happens in generated packers like xnn_x32_packw_gemm_goi_ukernel_x12__neon_ld4lane_u4, L20. Quantized dynamic GEMMs get a dedicated equivalent in xnn_pack_lh_f32_qdint8, L170, which pre-quantizes the left-hand side.

Repacking on every app start is wasteful. The weights cache solves this: xnn_internal_init_weights_cache, L281 creates a growable buffer, packed weights are written into it, and xnn_finalize_weights_cache, L514 seals it into a stable region that can be memoized and even serialized for the next launch. The underlying bytes come from xnn_allocate_weights_memory, L313, which allocates at page granularity and can drop write permission once packing is done — read-only packed weights are both safer and friendlier to the OS.

At run time, convolution-style operators use indirection tables instead of im2col: xnn_indirection_init_conv2d, L22 precomputes, for every output pixel and kernel tap, the pointer to the relevant input row. The kernels then just walk a table of pointers. xnn_invoke_runtime, L1238 walks the prepared operator list in topological order and each operator dispatches through function pointers like xnn_compute_gemm, which is where your float tensors finally meet SIMD.

Microkernels: Where the FLOPs Live

Every “compute” entry point ends in a microkernel — a fixed-tile function written in intrinsics or generated assembly. Selection is data-driven: init_hardware_config, L262 probes the CPU once (CPUID on x86, getauxval on Linux ARM); the resulting flags feed per-arch config tables such as xnn_init_f32_gemm_config, L7056, which chooses among a small forest of candidates. The naming encodes everything: f32-gemm-1x8-minmax-neon-dup-ld64 means fp32, 1×8 tile, min-max clamped, NEON with broadcast loads and 64-byte cache-line hints. Tile sizing is not arbitrary either — xnn_gemm_best_tile_size, L66 heuristically balances register pressure against weight reuse for small-M problems, which is exactly the regime mobile inference lives in.

Convolution gets direct kernels too: instead of lowering convolutions to GEMM, hot 3×3 cases run through specialized code like xnn_f32_conv_hwc_ukernel_3x3s2p1c3x8__aarch64_neonfma_2x2, L22, which processes the classic mobile conv directly in HWC layout, two output rows at a time. A look into the gen folders shows the scale: per-architecture .c and .S files for every tile × ISA × datatype combination, all produced by generator templates. Teardown is correspondingly tidy: xnn_delete_operator, L19 frees each operator through its type table, and the runtime releases the arena in one go.

From Install to Your First Model

XNNPACK is a C library, so “installation” means building it. The documented quick start uses CMake from the repo root (no official PyPI wheel exists, despite the abundance of xnnpack namesakes):

git clone https://github.com/google/XNNPACK.git
cd XNNPACK
cmake -B build -DCMAKE_BUILD_TYPE=Release -DXNNPACK_BUILD_TESTS=OFF
cmake --build build --config Release -j

scripts/build-local.sh wraps the same flow and adds a test run with CTest, and Bazel is supported equally well if that is your ecosystem. Link the resulting static library, include xnnpack.h, and a minimal fully-connected inference looks like this — the exact calls we verified in the tree above:

#include <xnnpack.h>

float input[8] = {/* ... */};
float filter[16], bias[4], output[4];  // 8 inputs -> 4 outputs

struct xnn_subgraph* subgraph = NULL;
struct xnn_runtime* runtime = NULL;
xnn_initialize(NULL /* default allocator */);

xnn_create_subgraph(2 /* external value ids */, 0 /* flags */, &subgraph);
const size_t in_dims[2] = {1, 8}, out_dims[2] = {1, 4};
uint32_t v_in, v_out;
xnn_define_tensor_value(subgraph, xnn_datatype_fp32, 2, in_dims,
                        NULL, 0 /* external id */, 0, &v_in);
xnn_define_tensor_value(subgraph, xnn_datatype_fp32, 2, out_dims,
                        NULL, 1 /* external id */, 0, &v_out);
xnn_define_fully_connected(subgraph, -INFINITY, INFINITY,
                           v_in, XNN_INVALID_VALUE_ID /* static filter */,
                           XNN_INVALID_VALUE_ID /* static bias */, v_out,
                           0 /* flags */);

xnn_create_runtime_v2(subgraph, NULL /* threadpool */, 0, &runtime);

For static weights you would instead define filter and bias values up front so the packer can process them at setup time. Setup binds real buffers, invoke runs, and teardown is two calls:

struct xnn_external_value ext[2] = {
    {0, input}, {1, output},
};
xnn_setup_runtime_v2(runtime, 2, ext);
xnn_invoke_runtime(runtime);
xnn_delete_runtime(runtime);
xnn_deinitialize();

The distinguishing trait versus, say, wrapping a model in TensorFlow Lite: you feel every tensor. Value IDs, dims, datatypes, and buffer bindings are all explicit. That is exactly what makes it such a predictable dependency for engines — and exactly what makes it tedious for humans. For a Python-flavored view of a runtime that makes different trade-offs on the same problem, see our earlier llama.cpp tour, and for how an engine composes providers around libraries like this one, see ONNX Runtime.

Honest Limits

• The C API is loud. Building a real network means dozens of xnn_define_* calls with explicit IDs and dimension arrays. It is a kernel-target API, not an authoring API — you rarely write it directly; you generate it or delegate to it.
• Inference only. There is no autodiff, no training path, no gradient of any kind. Quantization-aware tricks like fake-quant emulation are out of scope; you bring already-quantized weights.
• Packing is format-locked. Packed weights are only meaningful to the library version that produced them; upgrading XNNPACK invalidates caches built with xnn_finalize_weights_cache, L514. The cache is also per-runtime-family — do not expect one buffer to serve arbitrary shapes.
• Static shapes dominate. The fast path assumes fixed dims at build time; dynamic batch GEMM exists, but many rewrites and packings must be redone when shapes change, and xnn_reshape_runtime only covers a subset of cases cheaply.
• Platform list is long but finite. Accelerated paths cover x86 (SSE/AVX/FMA3/AVX2/AVX512 and VNNI/BF16 variants), 32- and 64-bit ARM (NEON, dotprod, i8mm, bfloat16), RISC-V RVV, Hexagon HVX, and WASM SIMD. Everything else falls back to portable scalar code, which is correct but many times slower.
• Threadpool coupling. Default parallelism assumes pthreadpool; wiring your own executor means threading it through every create call rather than a global setting.

None of these limits are accidents — they are the price of a library that treats a CPU’s cache hierarchy as the primary design constraint. That focus is why XNNPACK remains the default compute backend for on-device engines, and why its source still feels like a crash course in fast portable C: start at xnn_create_runtime_v4, L570, follow the packing into the gen folders, and you will never look at a matrix multiply the same way again.

Enjoyed this post? Never miss out on future posts by following us