Most inference frameworks we have toured on this blog — ONNX Runtime, OpenVINO, TVM — are desktop-first projects that were later coaxed onto phones. Tencent’s ncnn went the other way. It was born in 2017 for QQ and WeChat-style mobile workloads, written in C++03-friendly C++ with zero third-party dependencies, and it treats an Android phone’s octa-core ARM CPU as the primary target, with Vulkan GPUs as the optional fast path. If llama.cpp is the story of squeezing LLMs out of laptops, ncnn is the story of squeezing convnets into 2 GB of phone RAM in 20 milliseconds a frame.

The design bet is easy to state and easy to miss in the code: no dynamic allocation after warm-up, no hidden OS calls, no abstraction you cannot delete. The whole library is a graph of Layer objects that shuffle refcounted Mat tensors around, and every millisecond-critical kernel has hand-written NEON, AVX, and shader variants. In this tour we read the source the way its authors layered it: the Net and Extractor lifecycle, the two-file model format, the generated layer registry, the big.LITTLE threading story, and the Vulkan detour.

We have covered the training-side tooling before — PyTorch, JAX, MLX — and the serving side in pieces on vLLM and Triton Inference Server. ncnn occupies the opposite corner of that map: no Python at runtime, no server, one .param text file and one .bin weight file shipped inside your APK. Understanding how those two files become matrix multiplies is the whole point of this post.

Overview of ncnn: the Net/Extractor API, the Blob and Mat tensor model, the generated layer registry feeding per-ISA CPU backends and the Vulkan path, plus the PNNX converter that produces the model files.

Reading the overview from left to right:

• Net::load_param, L63 parses the text .param file and builds the layer and blob arrays; load_model, L68 fills in weights from the binary .bin.
• Net::create_extractor, L134 hands out an Extractor, and Extractor::extract, L220 lazily executes only the subgraph your requested output actually needs.
• Blob, L24 is just a producer and consumer layer index — the DAG is encoded in two ints per edge.
• Mat::elempack, L360 is the packing factor (1, 4, or 8 floats per SIMD lane group) that every kernel negotiates around.
• layer_registry_entry, L147 pairs each of ~80 layer types with a creator function; the tables themselves are generated per ISA.
• create_layer_cpu, L442 picks avx512, avx, rvv, msa, or plain C++ tables at runtime depending on the CPU it lands on.
• Convolution_arm winograd dispatch, L491 shows the per-shape kernel selection that makes ARM convolution fast.
• set_cpu_powersave, L3086 pins worker threads to the big or little cluster — a knob almost no desktop framework has.

Two Files, One Model

ncnn models ship as a pair. The .param file starts with the magic number 7767517 — checked at net.cpp L1546 — followed by a layer count, a blob count, and then one line per layer: an index-free row of type, name, input blob count, output blob count, and key=value parameters. Every row is parsed through ParamDict, a small typed key-value store whose getters (paramdict.h L37) return caller-supplied defaults for missing keys, which is how a single Convolution row survives format evolution across a decade.

Weights live in .bin, decoded by ModelBinFromDataReader::load at modelbin.cpp L82. Each weight block begins with a 32-bit flag that names its storage: 0x01306B47 marks fp16 pairs, 0x0002C056 marks quantized int8, and plain float32 blocks carry no flag at all. Because a DataReader abstracts the source (datareader.h L21), the same bytes can stream from a file, a memory buffer, or an Android asset without the layer code caring.

The tooling in tools/ keeps this format honest. pnnx.cpp, L287 drives PNNX, the PyTorch Neural Network Exchange converter that traces TorchScript graphs (far more faithfully than the retired ONNX export path — compare with our ONNX tour) and emits .param/.bin. ncnnoptimize, L2780 folds constants and batch norms, ncnn2mem, L481 packs the pair into C arrays for load_param(const unsigned char*)-style zero-IO loading, and the quantize folder’s ncnn2table, L2310 runs KL-divergence calibration to produce the thresholds ncnn2int8, L1098 needs to rewrite weights.

The Registry: One Class, Nine ISA Tables

The whole kernel story hangs off a single abstract base class. Layer (layer.h L20) declares load_param and load_model for its share of the two files, plus two inference entry points: forward (L95) for the general case and forward_inplace (L100) for layers that can reuse their input buffer. Two capability flags drive the scheduler: one_blob_only (L46) lets the runtime skip vector plumbing for single-input layers, and support_inplace (L49) is what makes ReLU-shaped ops nearly free.

What makes ncnn unusual is how implementations attach. CMake globs src/layer/arm/*.cpp, src/layer/x86/*.cpp, and so on, and stamps out parallel creator tables into layer_registry.h.in (L5): one per CPU ISA plus a Vulkan table. At load time, create_layer (layer.cpp L408) allocates the plain-C++ version and, if the net was built with Vulkan enabled, attaches a GPU twin to the same object. At runtime with NCNN_RUNTIME_CPU enabled, create_layer_cpu (L442) consults the detected ISA and swaps in the avx512 or rvv table — one binary, many CPUs, no dispatch in the hot loop. Registering your own layer type is one register_custom_layer call, with CustomBit (layer_type.h L13) keeping custom indices clear of the built-ins.

Look inside the ARM convolution implementation and the value of that structure shows. The layer’s setup pass at convolution_arm.cpp L117 pre-packs weights into the exact blocked layout its kernels consume; the forward pass at L324 then chooses between transformed-weight Winograd tiles (L491), an im2col+GEMM path (L562), and direct NEON loops by shape. The x86 sibling (convolution_x86.cpp L279) mirrors the same decisions for AVX and AVX512 packs. This is the same “unpack once, run many” philosophy behind GGML’s quantized kernels and CUTLASS’s tiled GEMMs, just aimed at 1x1 and 3x3 convolutions instead of attention.

Threads, Affinity, and the Big.LITTLE Bet

ncnn’s concurrency model is refreshingly blunt: OpenMP. Option::num_threads (option.h L38) is threaded down through every forward call, and kernels carve channels into _nT slices. The interesting part is what surrounds it. cpu.cpp probes the SoC into little, big, and all-cluster affinity masks, and set_cpu_powersave (L3086) moves the process onto one of them; get_big_cpu_count (L3037) then reports how many workers are actually worth spawning. On a phone where the little cores are 4x slower, running 8 threads across all cores is a latency loss — ncnn is one of the few frameworks that encodes that truth in an API.

Memory gets the same treatment. Blobs between layers are refcounted Mats (mat.h L347) — a data pointer, dims w/h/d/c, per-element packing in elempack, and padded channel stride cstep (L373). In light mode the Extractor recycles intermediates through a PoolAllocator (allocator.h L151), so steady-state inference performs no allocation churn, which matters both for latency spikes and for Android’s memory watchdog. Preprocessing lives beside the tensor type: Mat::from_pixels (mat_pixel.cpp L2429) converts NV21 camera frames straight into float tensors, and substract_mean_normalize (mat.cpp L1804) does mean-subtract and scale in one pass.

From Install to Your First Model

Building is a two-line CMake exercise: configure with -DNCNN_VULKAN=ON (or the Android NDK toolchain file) and install; there are no dependencies to hunt beyond OpenMP. The public API then fits on a postcard:

#include "net.h"

ncnn::Net net;
net.opt.num_threads = 4;          // Option knobs, see option.h L38
net.load_param("model.param");    // net.h L75
net.load_model("model.bin");      // net.h L95

ncnn::Mat in = ncnn::Mat::from_pixels(
    rgba_data, ncnn::Mat::PIXEL_RGBA2RGB, 224, 224);
const float mean[3] = {127.5f, 127.5f, 127.5f};
const float norm[3] = {0.007843f, 0.007843f, 0.007843f};
in.substract_mean_normalize(mean, norm);

ncnn::Extractor ex = net.create_extractor();  // net.h L134
ex.input("data", in);                         // net.h L214
ncnn::Mat out;
ex.extract("prob", out);                      // net.h L220

Three things are worth noticing. First, Extractor::extract is pull-based: it walks backwards from the named output through Blob::consumer/producer links and runs only the layers your output depends on, via the recursive forward_layer at net.cpp L138. Second, the same Net can serve many extractors concurrently; blob storage is per-extractor, weights are shared. Third, the error path is honest — extract a misspelled blob name and the runtime logs the exact ex.extract("name", out); lines you should have written, as the lookup failure branch in net.cpp L3194 shows. For a deeper contrast in execution models, see our tour of torch.compile’s runtime and of ExecuTorch, PyTorch’s own answer to on-device.

The Vulkan Detour

When a phone GPU is worth using, ncnn uses it — but on its own terms. Everything is compute-shader based: VkMat (mat.h L389) mirrors Mat with device-memory allocators, and VkCompute (command.h L22) is a reusable command buffer that layers record shader dispatches into. The Vulkan convolution twin sets up its shader bindings and weight staging at convolution_vulkan.cpp L71, and its GPU forward at L1759 records work for the whole graph to submit in one go — ncnn prefers fewer, larger submissions because phone driver overhead dwarfs desktop. Device capability probing (get_gpu_count, gpu.h L185) happens early, and the code is disarmingly pragmatic: at net.cpp L1579, any missing feature quietly flips use_vulkan_compute off and falls back to CPU mid-load. A net can even interleave — the GPU branch of forward_layer (L213) hops between CPU and GPU blobs per layer, staging through host memory when a layer has no shader.

Honest Limits

ncnn is an inference engine with strong opinions, and the edges show. There is no autograd and no training path — unlike tinygrad or PyTorch itself, every graph is frozen at conversion time, and dynamic control flow or data-dependent shapes must be flattened into the static layer list before deployment. The C++-only runtime API is a feature for APK size and a tax for experimentation; the Python bindings exist but are thin, so interactive debugging leans on the benchmark and squeezenet example tools rather than a REPL. Quantization is post-training only and needs a calibration dataset through ncnn2table; there is no QAT story, and quantized accuracy on depthwise-heavy mobile backbones needs the usual per-layer sensitivity massaging. LLM support is real but young: the Extractor grew kv-cache allocators (net.h L195) and the quantize folder gained LLM-specific tables recently, yet the ecosystem of ready-made transformer models is far thinner than llama.cpp’s or MLC-LLM’s. And the layer registry, for all its ISA tables, still expects you to write a shader or NEON intrinsics for any op the converters cannot decompose — extensibility by C++, not by config.

Those limits are the price of the pitch: a 1 MB library, no dependencies, deterministic latency, and kernels you can actually read. For on-device vision work — detection, segmentation, super-resolution, face chains at 60 fps — ncnn remains the reference point every mobile-first runtime is measured against, and its source is one of the best places to learn what “mobile-first” actually costs.

Enjoyed this post? Never miss out on future posts by following us