There is a particular kind of satisfaction in watching a multi-billion-parameter model generate fluent text on a laptop with no discrete GPU, fully offline, at usable speed. That experience is exactly what ggml-org/llama.cpp ships: LLM and vision-language inference written in plain C/C++ with no dependencies, MIT-licensed, and hand-tuned for hardware ranging from x86 servers with AVX512 to Apple silicon, RISC-V boards, and every GPU vendor with a working driver. The install has compressed to one line — curl -LsS https://llama.app/install.sh

sh on Unix or irm https://llama.app/install.ps1

iex on Windows — after which llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF downloads a quantized model straight from Hugging Face and starts chatting, and llama serve exposes an OpenAI-compatible HTTP endpoint with a built-in web UI. The README’s claim list is concrete: 1.5-bit through 8-bit integer quantization, custom CUDA kernels with HIP and MUSA support, Vulkan and SYCL backends, and CPU plus GPU hybrid inference for models larger than total VRAM.

The source organizes into four territories. The src directory holds the engine itself, one flat set of llama-prefixed modules where the public API, context scheduler, model definitions, compute graph builder, memory systems, and sampler chain live shoulder to shoulder. The ggml directory is the vendored tensor library, a project in its own right with a central math core and eighteen device backends from CPU to WebGPU. The tools directory carries the HTTP server, the multimodal helper, benchmarks, and a family of small single-purpose executables, while common holds shared argument parsing and logging. Ringing all of it are the format tooling that makes the ecosystem work: GBNF grammars, a Python GGUF library, and conversion scripts that turn Hugging Face checkpoints into the single-file model format. As always in this series, what follows is an educational tour of published source code.

llama.cpp at a glance: the C API drives the context, which builds compute graphs over pluggable memory and dispatches them through the ggml tensor library to hardware backends; the tokenizer feeds tokens in, the sampler draws tokens out, and the HTTP server plus multimodal helper wrap the whole engine.

Reading the overview from left to right:

• Everything starts from the C API, a five-hundred-line interface that bindings in every language compile against.
• Text enters through the tokenizer, four thousand lines of Unicode-aware segmentation for BPE, SentencePiece, and friends.
• The context owns a model instance and schedules every prefill and decode step.
• The model source defines layers for dozens of architectures, expressing each forward pass through the graph builder.
• Attention reads and writes state via the KV cache, and the sampler turns logits into tokens.
• ggml’s core is the tensor engine, dispatching through the backend registry.
• llama-server wraps the API in HTTP, with the multimodal helper adding images and audio.

Why You Need This

The first reason is coverage of hardware that no other single engine matches. The backend table in the README runs from BLAS and BLIS through CANN for Ascend NPUs, Hexagon for Snapdragon, IBM zDNN, OpenCL for Adreno GPUs, OpenVINO, RPC, SYCL, VirtGPU, WebGPU, and ZenDNN, with CUDA, Metal, and Vulkan as the flagship GPU stories — and the source backs the table, since ggml/src contains eighteen backend directories. The registry in ggml/src/ggml-backend.cpp enumerates devices, allocates buffers, and schedules work across them, which is what makes the CPU plus GPU hybrid claim real: tensors are placed per device, so a model whose weights exceed VRAM can still run with the hot layers on the GPU. At the bottom sits the ggml core, nearly seven thousand lines of operator dispatch and compute plans, and the quantized kernels, five thousand lines implementing every block format from Q1 through Q8 so a 4-bit model fits in a quarter of the memory and often runs faster than its full-precision original.

The second reason is that the engine itself is a masterclass in inference design, small enough to read end to end. The context is the conductor at nearly four thousand lines: it takes batches, sequences decoding, manages output, and drives the graph builder, which constructs one ggml computation graph per step with attention variants for every architecture. Memory is a plug-in contract in the memory interface: classic transformers use the KV cache with explicit cell management, Mamba-style models use the recurrent state, and architectures like Jamba mix both through the hybrid memory. Sampling is equally decomposed — the sampler chain at three and a half thousand lines stages penalties, temperatures, top-k, top-p, and grammar constraints, with GBNF defined in the grammar module forcing output to match a formal grammar when you need structured text.

The third reason is the serving and tooling layer, which turns the engine into a product. llama-server is a thin entry point over a modular service: the server context carries over five thousand lines that translate HTTP requests into inference tasks, the task queue orders concurrent requests, the streaming module emits server-sent events token by token, and the MCP bridge wires external tool servers into chat so the local model can call out to the world. Around it live the rest of the toolbox: the multimodal helper encodes images and audio through projectors, the argument parser defines four thousand lines of parameters shared by every executable, and the conversion scripts plus the Python GGUF writer keep the single-file model format flowing from Hugging Face checkpoints.

The detail view: the public API with tokenizer and grammar at the top, the runtime core with context, batch, graph, and sampler beside the memory systems, model loading with loader, architectures, adapters, and quantization on the left, the ggml core and hardware backends underneath, and the server modules with CLI and conversion tooling on the right.

Walking the detail diagram, the model-loading tier explains how a GGUF file becomes runnable state. The model loader memory-maps the file through the mmap helper, resolves architecture metadata against the architecture registry — a twelve-hundred-line catalog of tensor names and hyperparameters covering everything from LLaMA and Qwen to specialist families — and hands validated tensors to the model. LoRA adapters merge at runtime so one base model can wear many fine-tunes, and the quantization module can rewrite a model between block formats after the fact. On the backend tier, the CPU backend pairs SIMD kernels with a thread pool, the CUDA backend carries custom kernels at five thousand lines, the Metal backend ships precompiled shader fusion for Apple silicon, and the Vulkan backend is the single largest file in the repository at fourteen thousand lines — one person’s cross-vendor GPU story in code.

The server tier reads as a study in wrapping a synchronous engine for concurrent callers. Requests arrive at the server context, become tasks in the queue, and results leave through the streaming module as server-sent events; the chat module maintains conversational state, the schema module turns JSON Schema into GBNF grammars, and the MCP bridge connects external tool servers so a locally hosted model participates in agent workflows. The new CLI surface is deliberately small — the command dispatcher is fifty-four lines — because the README’s llama and llama serve commands are a friendly front door over the same engine every other tool uses directly.

From Install to the First Token

The install is one script and the program shape is: pull a GGUF model by name, start a context, submit a batch of prompt tokens, and stream sampled output. The same binary family covers the workflow end to end — llama tokenize and llama tts at the edges, llama quantize and the imatrix tool for making quantized weights, llama perplexity and llama llama-bench for measuring quality and speed, and llama server for the API. Structured output arrives by attaching a GBNF grammar or a JSON schema to a request, multimodal input by loading an mmproj projector alongside the model, and tool calling by pointing the server at MCP endpoints. Everything runs from the same quantized file format, which is why a model downloaded once plays identically on a phone, a laptop, and a GPU server.

Honest limits: this is an inference engine, not a training framework — fine-tuning is limited to LoRA tools, and anything beyond needs external projects. The C and C++ core is hand-managed memory throughout, which is the price of zero dependencies, and the API has historically moved between versions as the architecture list grows. Single-node is the story here; multi-machine inference is delegated to the RPC backend or external orchestrators. Model coverage depends on the conversion scripts keeping pace with the architecture zoo, so brand-new architectures land with a delay. But as a study in how to make large models small — block formats, backend registries, pluggable memory, a graph builder that reads like pseudocode — ggml-org/llama.cpp remains the most instructive inference codebase in this series, and the one that quietly powers most of the local AI movement.

Enjoyed this post? Never miss out on future posts by following us