The most consequential project in local inference has quietly moved house - the repository now lives as ggml-org/llama.cpp - and while the logo stayed, the internals have been rebuilt around an abstraction that barely existed when this series first toured the codebase: a polymorphic memory layer. Where older versions wired a single KV cache directly into llama_context, the current master defines llama_memory_i in src/llama-memory.h and implements it five ways - the classic attention cache, a dual sliding-window cache, recurrent state cells, a hybrid of the two, and more - so that transformer models, Mamba-style models, and everything in between flow through one decode loop. This second tour of the project skips what we covered before and goes straight for the redesign: the memory interface, the find-slot machinery that decides whether your prompt fits, and the one-shot optimization retry that stands between you and an error.

As always in this series, what follows is an educational tour of published source code.

Overview: the public API feeds a decode loop that initializes batches, asks a polymorphic memory module for context, and processes ubatches through the graph; the KV cache anchors five implementations; failures trigger a one-shot optimization retry; the modular server drives it all from a task queue.

Reading the overview from left to right:

• src/llama.cpp is now just 621 lines of C API exports - the real engine lives in llama-context.cpp, 4,719 lines, whose decode at line 1821 is the single path every model takes.
• llama_context::decode at line 1821 initializes the batch allocator at 1879, applies pending memory shifts at 1920, and enters a loop asking the memory module for a context via init_batch at 1925.
• src/llama-batch.cpp, 1,493 lines, is the traffic splitter: llama_batch_allocr::init at line 28 validates input, then split_simple at 585, split_equal at 619, and split_seq at 790 carve one user batch into ubatches the graph can chew.
• src/llama-memory.h is the new contract: init_batch at line 89, init_full at 95, init_update at 99, plus a seq_* family at 111-118 that gives every backend the same branching, copying, and position-shift vocabulary.
• src/llama-kv-cache.cpp, 2,962 lines, is the reference implementation: prepare at line 771 walks ubatches, find_slot at 918 hunts for contiguous space, and apply_ubatch at 1117 commits the choice.
• src/llama-kv-cache-iswa.cpp holds the SWA dual cache in 355 lines - a full cache for sinks and a sliding one for the rest - while llama-memory-hybrid.cpp, 283 lines, pairs attention and recurrent modules for Jamba-style architectures, and llama-memory-recurrent.cpp, 1,386 lines, keeps fixed-size state for Mamba-family models.
• src/llama-sampler.cpp, 4,391 lines, chains token filters around one softmax at line 293: greedy at 1113, dist at 1405, top_k at 1525, top_p at 1725, min_p at 1888, typical at 2000, temp at 2110, xtc at 2422, grammar at 2831, and dry at 3645.
• tools/server/server-queue.cpp, 621 lines, shows how far the server has modularized: post at line 28, deferred tasks at 76, and worker_loop at 163 - with a 6,313-line server-context.cpp, a 1,904-line server-task.cpp, and an MCP integration at server-mcp.cpp (799 lines) behind a main.cpp of just 5 lines.

Why You Need This

The polymorphic memory layer is the answer to a question this series keeps circling: how does one C++ codebase serve both a transformer and a state-space model without forking? The old answer was if-statements scattered through the context; the new answer is that llama_context holds a single llama_memory_i pointer and speaks only the interface - init_batch to start, seq_rm and seq_cp to edit, state_write at line 126 of the header to snapshot. When you run a Mamba model, the pointer points at llama-memory-recurrent.cpp; a hybrid Jamba architecture gets the 283-line coordinator; a Llama-family model gets the classic cache. Compare this with how vLLM bolted sliding-window support onto its paged allocator, or how ollama embeds an older llama.cpp and inherits whatever the snapshot had - reading the interface here shows what a clean second-architecture migration looks like when it is finished.

The find-slot machinery is worth its own visit because it is the heart of multi-sequence serving. find_slot at line 918 searches the cell grid for room for a ubatch - contiguity per sequence, headroom across the batch - and prepare at 771 runs it ahead of compute so failure is cheap. When preparation fails, decode’s status switch at 1930-1962 hits LLAMA_MEMORY_STATUS_FAILED_PREPARE and does exactly one thing: memory_update(true) at 1945, which shifts cache contents and optimizes fragmentation, then retries the same batch once - the debug line at 1946 says “retrying batch size after cache optimization,” and a second failure returns 1 at 1954. One retry, not a loop: the design accepts that a genuinely full context must fail loudly rather than thrash. Every LLM server reinvents this dance - llamafile sidesteps it with fixed allocations, TGI leans on its router’s queue - and seeing the minimal correct version in 30 lines is the best tutorial available.

Finally, the sampler file is a museum of decoding folklore, and the current layout teaches the chain-of-responsibility pattern with unusual clarity. Every filter - top_k, top_p, min_p, typical, temp, xtc, dry - is a tiny struct that mutates a token array in place, all funneling through the shared softmax at line 293, and the grammar sampler at 2831 simply zeroes out logits that the grammar forbids. The newest twist is backend sampling: decode opens a per-batch sampling transaction at 1974-1976 with llama_sampler_backend_begin, pushing the chain onto the GPU so the CPU never sees intermediate distributions - the same direction whisper.cpp took for its own hot loops. The sibling project comparison is not decorative: both live under the ggml umbrella, share ggml’s backend abstraction, and demonstrate how far a dependency-free C/C++ stack can stretch.

Detail: the decode loop wires batch splitting, memory preparation, graph building, and backend sampling; five memory backends share one sequence-operations vocabulary; the modular server queues tasks into the same C API.

The decode loop repays a full read. After the allocator initializes at 1879, decode counts outputs per sequence at 1853-1877 and rejects batches that exceed the backend-sampling output cap - a new constraint that did not exist when sampling lived on the CPU. The memory handoff at 1925 returns a llama_memory_context_ptr, a scoped object whose get_status() at 1930 tells the loop whether preparation succeeded; the one-shot optimize path lives at 1940-1955. Per ubatch, process_ubatch at line 1507 builds the graph via llama-graph.cpp (4,265 lines), computes, and on failure rolls back just the positions of that ubatch at 2004-2015 - an undo log in twelve lines. output_reorder at 2422 then untangles per-sequence outputs into batch order, and synchronize at 871 is the barrier callers hit before reading logits. The legacy llama_batch wrappers at 4601-4606 adapt the old public API onto llama_batch_ext, so years of downstream code keep compiling - a compatibility discipline worth copying in any long-lived library.

The KV cache itself is a small database, and the seq_* operations are its CRUD API. seq_rm at line 404 removes position ranges, seq_cp at 473 duplicates a sequence - the operation behind speculative decoding’s draft trees - seq_keep at 565 prunes everything else, and seq_add at 592 shifts positions, which is the mechanism behind context shifting when a conversation outgrows its window. prepare at 771 and find_slot at 918 enforce the invariants: cells for a ubatch must be contiguous per sequence, and get_can_shift at 1210 declares whether a backend can slide at all (recurrent state cannot; attention caches can). set_input_kq_mask at 1761 then writes the attention mask that encodes every one of those decisions into the graph - the last place where sequence bookkeeping becomes tensor data. The iswa variant at 355 lines runs two caches side by side because sliding-window layers and full-attention layers in the same model (Gemma-style interleaving) need different eviction horizons, and the hybrid coordinator delegates per layer type.

The server rewrite is the quiet headline of this tour. main.cpp is five lines; server.cpp holds 562; the actual behavior is spread across eighteen focused modules. server-queue.cpp posts tasks at line 28, defers continuation-style tasks at 76, and pops them when a slot frees at 90, while worker_loop at 163 drains the queue - a producer-consumer design that replaced the old monolithic loop. server-context.cpp (6,313 lines) owns inference slots and task execution, server-task.cpp (1,904 lines) defines the task types and result marshaling, server-mcp.cpp (799 lines) wires Model Context Protocol tool calling into completions, and the 5-line main is the reviewability payoff: a new contributor reads one module, not one monster. The same C API the CLI and bindings call is all the server uses, which keeps the library honest - if the server cannot do it through the public interface, users cannot either.

The model side stays the least changed. llama-model.cpp (3,985 lines) and the loader build tensor graphs per architecture at load time, llama-grammar.cpp (1,530 lines) implements the GBNF engine that the grammar sampler consults, and the whole stack sits on ggml’s backend abstraction - CPU, CUDA, Metal, Vulkan, SYCL - so the memory layer above never learns which hardware is underneath. That layering, interface over implementation over hardware, is the durable lesson of this codebase: the parts that were refactored (memory, server) are exactly the parts that had leaked across layers, and the parts left alone were already clean.

From Install to a Cached Sequence

Building and driving the server still takes three commands (abridged):

cmake -B build && cmake --build build --config Release -j
./build/bin/llama-server -m models/qwen2.5-7b-q4_k_m.gguf --ctx-size 8192 --port 8080
curl http://localhost:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}'
# observe the cache at work:
curl http://localhost:8080/props | head   # default_generation_settings, slots

Prompt caching is automatic: a follow-up request sharing the conversation prefix finds its cells via find_slot and computes only the new tokens - watch the timings field report n_prompt_tokens_processed shrinking to single digits.

Honest limits: the memory redesign is C++17 through and through, and the interface’s power comes with pointer juggling that demands care from embedders; the one-shot optimization retry means pathological fragmentation patterns can still fail a batch that a second retry would fit; recurrent memory keeps fixed-size state, so long-context claims for Mamba-family models depend on that state size, not on cache eviction; backend sampling caps outputs per sequence (enforced at decode 1853-1877), which unusual batch shapes can hit unexpectedly; the server’s 6,313-line context module is a lot of state in one file despite the modularization; and the org move to ggml-org means old clone URLs redirect - pin your remotes and your known_hosts.

Enjoyed this post? Never miss out on future posts by following us