Every deep learning framework makes the same bargain: you trade visibility for convenience, and after a few years you can ship models without ever seeing a training loop. llm.c, Andrej Karpathy’s follow-up to nanoGPT, tears up that bargain. It trains real GPT-2 and GPT-3 sized models in pure C and CUDA — the README’s opening line boasts that you need neither 245 MB of PyTorch nor 107 MB of CPython — and it does so under the MIT license in a repository you can read cover to cover in an afternoon. The mainline trainer is one file of 1,751 lines; the CPU reference is about a thousand more. Nothing between your data and your gradients hides behind an abstraction, and the porting energy the project unleashed says something too: the README’s forks section alone lists implementations in Rust, Zig, Go, Java, Swift, C#, Mojo, Metal, and OpenCL.

The project’s discipline is what makes it more than a stunt. Every C and CUDA path is paired with a PyTorch twin, and the two must agree: a debug-state file written by the Python side is loaded by the C tests, which compare logits, loss, and ten steps of Adam before declaring “overall okay”. The kernel library in dev/cuda starts with naive implementations and climbs to fast ones, each documented well enough to teach from, and a newer Llama 3 reference extends the same scaffolding to a modern architecture. Multi-GPU runs shard optimizer state with a ZeRO implementation in a single header, and the Makefile’s flags are the entire feature surface — precision, cuDNN attention, MPI versus NCCL. As always in this series, what follows is an educational tour of published source code.

The overview shows the Makefile fanning out into three trainers, the header-only components they share, and the PyTorch oracles that keep them honest.

Reading the overview from left to right:

• The mainline trainer is a single 1,751-line CUDA program that reproduces GPT-2 (124M) and GPT-3 model sizes with mixed precision.
• The CPU trainer is a pure C, fp32 reference in about one thousand readable lines — the best first file in the repo.
• The PyTorch twin is a lightly tweaked nanoGPT whose activations must match the C code exactly.
• The Makefile is the whole feature surface: flags for precision, cuDNN attention, and multi-GPU support.
• The dataloader header memory-maps plain uint16 token bins, no dataset library required.
• The matmul header wraps cuBLAS and cuBLASLt, the one place where vendor libraries are allowed to dominate.
• The ZeRO header shards optimizer state across ranks so multi-GPU runs actually scale.
• The kernel library collects hand-written, heavily documented kernels that climb from naive to fast.

Why You Need This

The first reason is education through subtraction. In a framework, the backward pass is a function you call; in llm.c, it is several hundred lines of explicit C that you can step through with a debugger, and the README even tells you to replace -O3 with -g and do exactly that. Reading the CPU trainer end to end teaches you where every tensor lives, when gradients flow, and what an optimizer update actually touches — knowledge that transfers back to PyTorch because you finally know what it is doing for you. Any question about the backward pass becomes a breakpoint you can actually set rather than an equation you take on faith, which is a different kind of learning. The hand-written backward pass is the real prize: backpropagation stops being a magic word and becomes ordinary calculus applied layer by layer, with the chain rule written out in plain code. The repo even ships a layernorm tutorial that builds one GPT-2 layer step by step as a gentle on-ramp.

The second reason is the verification culture. Machine learning code is famously hard to test, and this repo shows the workaround: keep a PyTorch twin, export a debug state, and make the C side prove agreement on logits, loss, and ten Adam steps before anything ships. The same pattern repeats below the trainer: dev/test holds unit tests for the dataloader and other components, the test binaries exist in fp32 and mixed-precision flavors, and the reproduction scripts tie the whole thing to published model sizes. The tests compare against saved tensors, so a kernel change that shifts one bit of the math fails loudly instead of silently degrading a week-long run. If you maintain numerical code anywhere — quantization, inference servers, training jobs — this dual-implementation habit is the single most valuable thing to steal from the project, because it converts “it seems to work” into a bit-exact comparison you can rerun any time the code changes.

The third reason is performance engineering with its receipts attached. The attention kernel file walks twelve hundred documented lines from a naive implementation to fast variants, which turns “Flash Attention is faster” from folklore into something you can watch happen. The MFU header measures model flops utilization against hardware peak so speed claims have a unit, the matmul header shows exactly where vendor libraries earn their keep, and the README states the complexity budget plainly: a two percent win that costs five hundred lines may be rejected, while making cuBLAS the default matmul backend is welcomed because it is one interpretable line and a common dependency. Between the benchmarked kernels, the measured utilization, and that stated budget, the speed story closes its loop — claims, measurements, and defaults all point the same way. Watching that trade-off enforced in code review is an education in itself.

The detail graph separates six bands: root programs, PyTorch reference, llmc components, the kernel library, data preparation, and launch scripts.

Start at the root programs. The Makefile builds all the binaries, and the include lists tell the architecture story: the mainline trainer pulls in a dozen llmc headers, while the CPU trainer needs only utilities, tokenizer, and dataloader — the same loop, no CUDA anywhere. The frozen fp32 trainer preserves an early checkpoint of the GPU code for readers who want the simpler path, the test binary exists purely to certify agreement with Python — it loads a debug state written by the PyTorch twin, runs a forward pass plus ten Adam iterations, and prints “overall okay” only when the losses match — and a profiling program alongside them keeps the speed story measurable rather than anecdotal.

The components band is where the loop gets its organs. Batches flow from the dataloader, which expects bin files with a 1,024-byte header and a stream of uint16 tokens. Vocabulary handling lives in the tokenizer header, learning-rate warmup and decay in schedulers, and generation after training in the sampler, with a logger and an outlier detector keeping long runs observable. The layer headers each pick a strategy: attention compiles to the manual kernels or cuDNN Flash Attention depending on a Make flag, layernorm fuses the residual add, GELU ships its own forward and backward kernels, and AdamW fuses the update. For multi-GPU, global norm computes cross-rank clipping and ZeRO splits the optimizer state so memory scales with the cluster instead of the model.

The remaining bands complete the ecosystem. The kernel library pairs every forward file with a backward sibling — attention backward mirrors its forward, layernorm backward at 1,390 lines is the largest in the repo — and the fused classifier merges output projection with softmax loss. The softmax kernels show the numerical trick behind stable attention, and matmul backward completes the backward set with fused bias variants. On the data side, tinyshakespeare.py downloads and tokenizes into the exact bin format the dataloader expects, with sibling scripts for larger corpora, and the starter pack script fetches model, tokenizer, debug state, and data in one shot. Launch scripts pin the hyperparameters: run_gpt2_124M.sh for one node, siblings for the 350M, 774M, and 1,558M sizes, the MPI launcher for multi-node NCCL runs, and a notebook for turning sweep logs into loss curves.

From Install to First Shakespeare

The CPU quickstart needs no GPU at all:

chmod u+x ./dev/download_starter_pack.sh
./dev/download_starter_pack.sh
make train_gpt2
OMP_NUM_THREADS=8 ./train_gpt2

This fetches the GPT-2 124M weights, tokenizer, debug state, and tokenized tinyshakespeare, then finetunes for forty steps with AdamW and samples Shakespeare-flavored text as validation loss falls — the README’s sample run starts near 5.25 and ends near 3.97. The same starter pack powers the fp32 GPU path (make train_gpt2fp32cu) and the mainline mixed-precision build (make train_gpt2cu), with USE_CUDNN=1 swapping in cuDNN Flash Attention if you have the libraries installed. For a real reproduction run you point the trainer at a tokenized dataset and launch through the scripts, scaling out with mpirun -np <gpus> ./train_gpt2cu once MPI and NCCL are installed; the README even demonstrates a four-GPU learning-rate sweep driven by screen sessions, a reminder that at this level an experiment manager can be a shell loop. Reproducing GPT-2 124M end to end is the project’s own recommended introduction, with the full walkthrough pinned in its discussions.

Honest limits: this is a pretraining research codebase, not a serving stack — there is no REST API, no quantized inference, no chat surface, and its speed comparisons are hardware-dependent claims against a moving PyTorch nightly rather than guarantees. Scope is deliberately GPT-2 and GPT-3 shaped, with Llama 3 covered only on the PyTorch side so far. CUDA is the first-class citizen and AMD support lives in forks, enabling cuDNN attention costs about a minute of compile time, and the multi-node scripts assume Linux cluster tooling like slurm and MPI. None of that diminishes the point: if you have ever wanted to watch a transformer learn with nothing hidden, this repository remains the clearest window in open source.

Enjoyed this post? Never miss out on future posts by following us