When we toured llama.cpp, the model layer got the attention — templated attention, KV-cache tricks, speculative decoding. Underneath all of it sits a separate repository that does the actual arithmetic: ggml, the tensor engine written in C that llama.cpp consumes as a library. The same engine also powers Whisper.cpp, Stable Diffusion ports, and a long tail of on-device AI projects, which makes it arguably the most executed piece of inference code on consumer hardware today. If ONNX is the contract for describing models, ggml is one of the most battle-tested answers for running them — without a framework, without Python, and often without a GPU.
The design is refreshingly blunt. One public header declares a vocabulary of tensor ops. A core library builds those ops into a compute graph and plans its memory. A scheduler splits the graph across whatever devices are registered — CPU always, plus CUDA, Metal, Vulkan, SYCL, ROCm and friends. Each backend implements the same op set against its own hardware, with the CPU backend carrying an entire sub-discipline of quantized matrix kernels. Around that sits GGUF, a file container that moves weights from disk to tensors, and a small optimizer for people who want training too.
The map below follows that flow: headers to core, core to allocator and scheduler, scheduler out to three real backends, and the GGUF reader feeding the whole thing.
Overview of ggml: ggml.h declares the op vocabulary; ggml.c builds compute graphs, ggml-alloc.c plans their memory, and the backend registry hands devices to the scheduler, which splits work across the CPU, CUDA, and Metal backends; GGUF files provide quantized weights from disk.
Reading the overview from left to right:
- The public surface is ggml.h, L1482, where
ggml_mul_matand hundreds of siblings are declared, plus gguf.h, L84 for the container. - The C core turns those calls into a graph: ggml.c, L3362 constructs each op node and ggml_build_forward_expand, L7359 grows the cgraph from an output tensor.
- Memory is planned before any kernel runs: ggml-alloc.c, L498 creates a graph allocator that packs tensors into buffer chunks.
- The scheduler takes over at ggml-backend.cpp, L1214, using the device registry (L119) to split the graph across the CPU backend, L3391, CUDA, L1950, and Metal, L550.
- Weights arrive through gguf_init_from_file, L1017, which parses the GGUF header and tensor table into tensors the graph can consume.
One Header, A Whole Compute Vocabulary
ggml’s API is deliberately C99, and it reads like a dialed-down PyTorch: you allocate a context with a fixed memory pool, create tensors, and chain op constructors. ggml_new_tensor, L836 is the root — everything else is shaped metadata around it. The interesting move is that op constructors like ggml_mul_mat, L1482 don’t compute anything; they allocate a result tensor whose src[] pointers point at the operands. The program you write in ggml is therefore a lazily built DAG, captured in the exact order you call the constructors.
The implementation side in ggml.c, L3362 is mostly bookkeeping — reshape checks, type promotion, op params — but the graph-building half deserves attention: ggml_build_forward_expand, L7359 walks parents depth-first into a flat array of nodes. Execution later is just “for each node in order, dispatch to the backend.” No closures, no VM, no tracing pass — the flat cgraph is the intermediate representation, and it is one of the reasons ggml binaries stay small enough to ship in a phone app.
Memory First: The Graph Allocator
Allocating one buffer per tensor would be fatal for transformer graphs with thousands of nodes, so ggml plans memory like a compiler register-allocates. ggml_gallocr_new_n, L498 builds an allocator over a list of buffer types, and the planning pass walks the graph assigning each tensor an offset inside a chunk — reusing the same bytes for tensors whose lifetimes never overlap. The dynamic allocator (tallocr_chunk, L115) keeps free blocks in size-ordered lists so reuse is a search, not a scan.
The output of that plan is a single ggml_backend_buffer per device, plus a schedule of set/cpy operations. This is invisible in the happy path but explains two things you notice in practice: peak memory is stable across runs (the plan is deterministic), and llama.cpp can report exact VRAM requirements before loading — the answer is simply the size of the plan.
The Scheduler: One Graph, Many Devices
The registry is where hardware enters. ggml-backend-reg.cpp, L119 constructs a static list of backends — CUDA (L121), Metal (L124), Vulkan, SYCL, and more — guarded by which libraries were compiled in. The CPU backend is always present, which is why every ggml program works on a bare laptop.
The scheduler then does the puzzle: ggml_backend_sched_split_graph, L1214 walks the graph and assigns every node to a device, honoring where weights physically live and where inputs are produced. The result is a sequence of splits; ggml_backend_sched_compute_splits, L1865 runs each split on its backend and copies boundary tensors between them. Compare this with IREE, which solves the same placement problem at compile time in MLIR: ggml does it at runtime in plain C, in a few hundred lines, favoring robustness over optimality — and for the mixed CPU/GPU rigs most people actually run, robustness is the right call.
The CPU Backend: Where Quantization Lives
The CPU backend is the oldest and most instructive. Execution begins at ggml_graph_compute, L3391, but the planning matters more: ggml_graph_plan, L2811 walks every node, asks each for its scratch requirements, and sums them into a work buffer plus a threadpool configuration. The dispatch (ggml_compute_forward, L1740) is a giant switch over op types, and matmul — the op that matters — lands in ggml_compute_forward_mul_mat, L1255, which fans rows across threads and calls the inner dot product. Standalone kernels like ggml_compute_forward_add, L654 live in their own file now, evidence of the repo’s slow split from one monolith into modules.
The inner loops are the real story. vec.h, L71 declares ggml_vec_dot_f32 and friends — the SIMD atoms, specialized per architecture in the arch directories. On top sit the quantized formats: quantize_row_q4_0_ref, L113 and quantize_row_q8_0_ref, L276 define block layouts (scales, minima, packed nibbles) that make a 4-bit weight look like a tiny struct of 32 elements. The trick ggml popularized is asymmetric quantization: keep weights at Q4 but requantize activations to Q8 on the fly, so the dot product is pure integer SIMD. When a weight layout benefits from reordering, repack.cpp interleaves blocks into GEMM-friendly tiles with dedicated GEMV kernels (L843) — the same tile-shape obsession you see in FlashInfer, just aimed at SIMD instead of tensor cores. And for servers with Sapphire Rapids silicon, the AMX path (ggml_amx_init, L214) requests tile-data permissions and provides its own buffer type — the technique KTransformers leans on for its CPU-heavy MoE offload.
GPU Backends: CUDA and Metal
The CUDA backend mirrors the CPU structure but swaps the hard parts. Device enumeration happens once in ggml_cuda_init, L265; op dispatch lands in ggml_cuda_mul_mat, L1950, which chooses between three regimes: for a single token against quantized weights, the vectorized kernel in mmvq.cu, L601 dequantizes in registers; for larger batches, ggml_cuda_mul_mat_q, L134 requantizes activations to Q8 and runs tiled integer matmuls from shared memory; for dense FP16 it falls back to tensor-core GEMM. Attention is separate and equally tiered — ggml_cuda_flash_attn_ext, L757 picks between an MMA variant for long contexts and a scalar vector variant for short ones. The graph-level entry (L4684) wraps all of it and, when the graph is safe, replays it on a CUDA graph to shave launch overhead.
Metal follows the same shape with Apple flavor: ggml_backend_metal_graph_compute, L550 encodes kernels onto command buffers, while ggml-metal-common.cpp, L35 hosts the kernel-choice heuristics — when a matmul is small enough for the matrix-accelerator path (L143) versus a plain SIMD path. Reading CUDA and Metal side by side is the best way to see what a “backend” really is: the same op list, two entirely different engineering cultures.
GGUF and the Training Loop
The container is what makes the ecosystem shareable. gguf.h, L84 declares the reader; gguf_init_from_file, L1017 (with the I/O in init_from_file_ptr, L966) parses a magic number, a version, a key-value metadata section, and a tensor table of (name, type, dimensions, offset) records. Weights are memory-mapped or read straight into buffers typed by the same block layouts from the quantization section — which is why a Q4_K_M GGUF produced by one tool loads identically everywhere. Contrast with the protobuf world of ONNX: GGUF bet on a simple binary footer and mmap-friendly layout, and for single-file LLM weights that bet has largely won.
Less known: ggml ships training too. ggml_opt_init, L549 builds static forward and backward graphs for a dataset shape, and ggml_opt_epoch, L881 runs the train/eval loop with callbacks — the machinery behind llama.cpp’s LoRA fine-tuning tools. Gradient support flows through the same graph builder we saw earlier (L7359), with each op carrying an optional backward function.
From Install to Your First Model
ggml builds with plain CMake — no Python, no package manager beyond your compiler:
git clone https://github.com/ggml-org/ggml
cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release
And a minimal C program computes y = W·x end to end — graph, allocator, scheduler, backend:
#include <ggml.h>
#include <ggml-alloc.h>
#include <ggml-backend.h>
#include <stdio.h>
int main(void) {
struct ggml_init_params ip = { 2 << 20, NULL, true };
struct ggml_context * ctx = ggml_init(ip);
int64_t nw[2] = {2, 2}, nx[1] = {2};
struct ggml_tensor * w = ggml_new_tensor(ctx, GGML_TYPE_F32, 2, nw);
struct ggml_tensor * x = ggml_new_tensor(ctx, GGML_TYPE_F32, 1, nx);
struct ggml_tensor * y = ggml_mul_mat(ctx, w, x);
struct ggml_cgraph * gf = ggml_new_graph(ctx);
ggml_build_forward_expand(gf, y);
ggml_backend_t cpu = ggml_backend_cpu_init();
float wd[4] = {1, 2, 3, 4}, xd[2] = {1, 1};
ggml_backend_tensor_set(w, wd, 0, ggml_nbytes(w));
ggml_backend_tensor_set(x, xd, 0, ggml_nbytes(x));
ggml_backend_graph_compute(cpu, gf);
float out[2];
ggml_backend_tensor_get(y, out, 0, ggml_nbytes(y));
printf("%f %f\n", out[0], out[1]); // 3.0 7.0
ggml_free(ctx);
return 0;
}
Twenty lines, and every layer of the map is exercised: context and tensors from ggml.h, graph building from ggml.c, allocation through the no-alloc context, execution through the CPU backend. Swap ggml_backend_cpu_init() for a CUDA device from the registry and the same graph runs on a GPU — that symmetry is the whole point.
Honest Limits and Where ggml Fits
Limits, honestly stated. ggml is a tensor library, not a compiler: there is no kernel fusion across ops, no static shape analysis, no autotuning search — kernels are hand-picked by heuristics, which is why exotic architectures sometimes lose to less famous stacks. The op set is shaped by transformer serving; convolutions and 2D vision ops exist but are not the center of gravity. Quantization is a zoo by now — Q4_0 through Q8_0, K-quants, I-quants — and choosing among them requires llama.cpp wiki knowledge that the ggml repo itself only partially documents. And the C API, while stable in practice, moves with llama.cpp’s needs; it is a library evolving inside a product, not a spec committee’s deliverable like ONNX.
For fit: read ggml when you want to understand what inference actually costs — a matmul is rows times threads times dot products, no more — or when you need to embed inference in something with no Python runtime. It also completes the mental model of our other tours: ONNX Runtime and Triton Inference Server show the serving view, MIGraphX and IREE show the compiler view, and ggml shows the machine view. All of them, plus the rest of the inference catalog, eventually reduce to what ggml does with zero ceremony: build a graph, plan the memory, split it across devices, and multiply. Enjoyed this post? Never miss out on future posts by following us 
