Most production LLM serving stacks are enormous: graph compilers, paged KV caches, continuous batching, speculative decoding. Google’s gemma.cpp walks in from the opposite direction. It is a lightweight, standalone C++ inference engine built for exactly one job — running the Gemma family of models (Gemma 2, Gemma 3, PaliGemma 2) on a CPU — and it deliberately keeps the core small enough to read in an afternoon. The README positions it as a descendant of the vertically-integrated, single-purpose engine tradition alongside ggml and the C-based mini-engines it explicitly cites; if you have followed llama.cpp, you already know the genre, but gemma.cpp’s execution is distinctly its own.

Why tour this one? Because it is a rare, honest boundary object between research and deployment. The authors say the quiet part out loud: deployment-oriented C++ runtimes are not designed for experimentation, while Python research frameworks compile away the low-level computation you actually want to study. gemma.cpp fills that gap with roughly 2K lines of core code plus ~4K lines of utilities, one flat directory tree, and no hidden layers of abstraction. You can trace a generated token from the SentencePiece vocabulary, through the decoder loop, into hand-tuned SIMD dot products, and back out to your terminal without ever leaving the repo.

This post walks that path in the order the code executes. The two diagrams below map the whole flow — first the coarse structure, then the detailed call graph.

Overview of gemma.cpp: the CLI drives the Gemma facade, which loads tensors from a single .sbs blob and runs prefill/decode through Highway-backed SIMD matmul kernels.

Reading the overview from left to right:

• CLI entry: main, L297 parses the command line, then hands off to the interactive ReplGemma, L92 loop that wraps each prompt and streams tokens back to the terminal.
• Facade: Gemma, L233 is the single public class. Its constructor reads weights, config, and tokenizer from one blob; its GenerateBatch, L266 method is the batched entry point.
• Forward pass: Gemma::Generate, L663 dispatches (via Highway’s runtime ISA selection) into the prefill and decode loop in gemma.cc, which calls one TransformerLayer, L83 per model layer.
• Attention: GemmaAttention, L342 computes QKV projections, applies PositionalEncodingQK, L75 rotary embeddings, and reads and writes the KVCache, L30 — a single flat float matrix.
• Config: every dimension comes from ModelConfig, L336, deserialized from the blob rather than hard-coded.
• Weights: WeightsPtrs, L273 walks all tensors via a name registry and pulls them from the ModelStore, L50 blob container.
• Kernels: MatMul, L1060 is where cycles go to work — autotuned, tiled, and parallelized across cache clusters.
• Sampling: each decode step ends in FusedSoftmaxAndSampleTopK, L1376, a fused softmax-plus-top-k routine with temperature support.

A ~2K LoC Core With a Serious Kernel Layer

The project advertises a “~2K LoC core implementation,” and that is roughly true for the model logic: configs, weights, activations, attention, and the generation loop. What the headline number hides is the quality bar of the supporting layers. The ops/ directory is not a toy. ops/matmul.h alone carries a cluster-aware parallelization strategy (MMParallelHierarchical, L157), a packed 32-byte MMConfig, L425 describing tile sizes per matrix shape, and an MMAutoTune, L504 state machine that times candidate configurations and disqualifies any that lands more than 25% behind the best.

That combination — tiny model code, industrial kernels — is the whole design philosophy. Compare it with llm.c, which keeps even the kernels simple in exchange for readability, or with ggml’s graph-based approach where ops live in a shared backend library. gemma.cpp instead specializes each kernel per model dimension at compile time and lets runtime autotuning pick the winner. The cost is complexity concentrated exactly where CPUs need it; the benefit is a genuinely fast CPU engine that still fits in your head.

The SIMD story runs through Google Highway, which lets one source file compile for AVX2, AVX3, Zen4, NEON, and an EMU128 fallback, choosing the target at process start. Where other engines hand-write per-ISA paths, gemma.cpp writes one vector-length-agnostic implementation and instantiates it per target via HWY_VISIT_TARGETS — you can see the pattern in the attention declarations and the matmul static dispatch files.

Weights, Blobs, and the .sbs Format

Model files use the .sbs extension, and the container behind it is refreshingly plain: BlobReader, L51 reads a header of hashed keys, builds a hash map from key to byte range, and hands out offsets. BlobWriter, L118 does the reverse with parallel I/O and an appended footer. No protobuf, no zip archives — just aligned byte spans addressed by name.

On top of that sit three layers of meaning. First, TensorInfoRegistry, L91 maps tensor names like qkv1_w or gating_ein to shapes derived from the config. Second, WeightsPtrs, L273 enumerates every tensor it needs via a ForEachTensor visitor, marking each as required or optional, and its ReadFromBlobs, L408 method picks between three read modes: decompress to BF16, read with padding, or memory-map the file directly. Third, a Fixup pass reshapes file layout into the layout the kernels want — for example, splitting stacked QKV or FFN gating matrices into separate w1/w2 tensors.

The compression formats are where gemma.cpp departs from the ordinary. Alongside BF16 and F32 it ships three custom schemes declared in SfpStream, L83 (an 8-bit “switching floating point” hybrid that picks between e4m3 and e5m2 characteristics per value), I8Stream, L93 (int8 groups of 128 with a shared scale), and NuqStream, L129 (non-uniform 4-bit quantization that clusters values via dynamic programming). Because decompression is fused into the dot products themselves — Decompress2, L583 produces SIMD vectors on the fly — a compressed model is not just smaller on disk, it is faster to run, since the bottleneck is memory bandwidth. The format philosophy resembles safetensors (flat, safe, mmap-friendly) but with compression built into the container instead of left to the model zoo.

The Forward Pass, Layer by Layer

Generation splits into two regimes. Prefill, handled by PrefillTBatch, L188, processes the whole prompt token-batched: up to 256 tokens flow through each layer at once, which amortizes weight loading across the batch. Decode, handled by GenerateT, L488, generates one token per query per step, batching across independent queries (decode_qbatch_size defaults to 16). The same TransformerLayer, L83 body serves both; only the batch axis changes.

Each layer runs the standard Gemma recipe: pre-attention RMSNorm, QKV projection, Rope, L358 rotary position embeddings (with the Gemma-specific twist of rotating dimension pairs i and i + d/2 rather than consecutive pairs), dot-product attention with logit capping for Gemma 2, a weighted sum over V, and a GeLU feed-forward network with the gated w1 ⊙ w2 structure. Attention reads its keys and values straight out of the KVCache, L30 matrix — kv_heads * qkv_dim * 2 columns per layer, one row per sequence position — and writes the new entry at the current position as it goes.

Extensibility is wired in through callbacks rather than subclasses. RuntimeConfig, L114 carries a stream_token function invoked for every generated token (returning false stops generation), an optional accept_token filter for constrained decoding, an optional sample_func to replace top-k entirely, and observers that see every layer’s activations. The bundled examples/hello_world uses exactly this to reject a user-specified set of token IDs mid-generation. Sampling itself defaults to FusedSoftmaxAndSampleTopK, L1376: softmax and top-k in one pass, with temperature, optionally deterministic via a seeded AES-CTR engine. If you have read our tokenizer deep dive, the SentencePiece integration will feel familiar — WrapAndTokenize, L80 applies the start_of_turn / end_of_turn chat template before any tensor math happens.

SIMD MatMul: Where the Cycles Go

Every projection in the model funnels through one entry point. MatMul, L1060 receives A (activations), B (weights, possibly still compressed), and a per-shape MatMulEnv, L677 that owns scratch buffers, per-shape autotuning state, and the thread pools. From there the kernel tiles C into small register-resident blocks — the constants kNR = 4 and kMaxMR = 4 keep FMA units fed — and streams over K and N in cache-sized panels.

The autotuner deserves a closer look because it encodes a real engineering trade-off. On first encounter with a matrix shape, MMCandidates, L496 generates dozens of plausible configurations: tile sizes mr/mc/kc/nc, loop order (parallelize N first, or M-and-N blocks together), and task granularity. The engine then literally benchmarks them against each other during inference, spending the first few calls finding the winner, and caches the result keyed by shape. The README’s advice that “the second and especially third query will be faster” is this machinery at work. Type specialization is layered on top: matmul_static_bf16.cc, matmul_static_sfp.cc, matmul_static_i8.cc, matmul_static_nuq.cc, and matmul_static_f32.cc each compile the full kernel set for one weight type, with MatMulStatic, L29 as the generated overload per ISA target.

Numerics are treated with unusual care for a hobbyist-scale project. The ConditionNumber, L51 helper in the dot product code estimates how many mantissa bits a given weight/vector pair actually needs, and error-compensated cascaded sums keep accumulation accurate. This is a nice reminder that inference quality on compressed weights is not luck — it is measured, Dot, L421 by Dot, L414.

From Install to Your First Model

The build is plain CMake with FetchContent pulling four dependencies: Highway, SentencePiece, nlohmann/json, and cpp-httplib. On Unix:

git clone https://github.com/google/gemma.cpp.git
cd gemma.cpp
cmake --preset make
cmake --build --preset make -j8

On Windows, use cmake --preset windows instead (the README recommends WSL or the Visual Studio Build Tools with the Clang toolset). Bazel users can bazel build -c opt --cxxopt=-std=c++20 :gemma.

Weights come from Kaggle: pick the “Gemma C++” variation of gemma-2, consent to the license, download, and extract with tar -xf archive.tar.gz. Start with gemma2-2b-it-sfp — the 8-bit switching-float variant is half the size of BF16 and faster to run. Then:

./gemma --tokenizer tokenizer.spm --weights gemma2-2b-it-sfp.sbs

(Models in the newer single-file format embed the tokenizer, so you can drop the --tokenizer flag entirely; io/migrate_weights converts older checkpoints.)

To embed the engine rather than chat with it, the pattern fits in one screen — this is a condensed version of the repo’s examples/hello_world/run.cc:

#include "gemma/gemma.h"
#include "gemma/gemma_args.h"
#include "util/threading_context.h"

int main(int argc, char** argv) {
  gcpp::LoaderArgs loader(argc, argv);      // --weights, --tokenizer
  gcpp::InferenceArgs inference(argc, argv);
  gcpp::ThreadingContext ctx;               // thread pools, caches
  gcpp::MatMulEnv env(ctx);                 // autotuning + scratch
  gcpp::Gemma gemma(loader, inference, ctx);
  gcpp::KVCache kv_cache(gemma.Config(), inference, ctx.allocator);

  const std::vector<int> tokens = gcpp::WrapAndTokenize(
      gemma.Tokenizer(), gemma.ChatTemplate(),
      gemma.Config().wrapping, 0, "Write a greeting to the world.");

  gcpp::RuntimeConfig rc = {
      .max_generated_tokens = 1024,
      .temperature = 1.0,
      .verbosity = 0,
      .stream_token = [](int token, float) { /* print */ return true; },
  };
  gcpp::TimingInfo timing;
  gemma.Generate(rc, tokens, 0, kv_cache, env, timing);
}

Five objects, one call. For comparison, standing up a full TGI container to serve the same model involves considerably more moving parts — which is precisely the trade gemma.cpp is making in reverse.

Honest Limits

Credit where due: the README does not oversell, and neither should we. Here is what this engine genuinely is not.

• CPU only. There is no CUDA, Metal, or Vulkan path. Everything runs through Highway’s CPU SIMD. If you need GPU throughput, look elsewhere; this is a research and embedding tool, not a throughput machine.
• A gated, narrow model menu. Weights must be downloaded through Kaggle’s consent flow, and conversion from other sources is only partially public — the safetensors converter is described as “not yet open sourced.” The supported set is Gemma 2, Gemma 3, and PaliGemma 2; fine-tunes require converting to .sbs yourself.
• The “tiny” label is load-bearing but selective. The 2K-LoC claim covers the model core; add the matmul/autotuning layer, the compression library, IO, and threading utilities and the real surface is closer to 6K lines — plus whatever you need to learn about Highway. Simplicity here means “one flat repo,” not “naive code.”
• Experimental edges. CLI usage is flagged experimental in the docs, long sequences get quadratic-attention slow, and while an SSE-style HTTP server binary now ships alongside the CLI, this is not a hardened multi-tenant serving stack. Treat it as the readable reference implementation that production systems — and other tutorials — can be checked against.

Those limits are also the point. gemma.cpp is one of the few places where you can read an entire inference engine — weights on disk, decompression, tiling, autotuning, sampling — without a guided tour. This post was that tour; the diff to your own fork is the next step.

Enjoyed this post? Never miss out on future posts by following us