Most “minimal GPT” repositories are minimal the way a demo reel is minimal: the training script is a toy, and everything real is hidden in a framework you must first learn. karpathy/nanoGPT refuses that bargain. It is a rewrite of minGPT that, in the author’s own words, prioritizes teeth over education, meaning the code stays readable but the numbers are real: train.py reproduces GPT-2 (124M) on OpenWebText on a single 8xA100 40GB node in about four days, landing at a validation loss near 2.85. The entire model lives in a 330-line model.py; the entire training loop lives in a 336-line train.py. There is no configuration framework, no trainer abstraction, no plugin system. As always in this series, what follows is an educational tour of published source code.

The repository wears its smallness proudly. A MIT license, two notebooks studying scaling laws and transformer parameter counting, a bench script, a sampling script, three data preparation folders, and a handful of config files. A November 2025 note at the top of the README now points newcomers to its successor nanochat and calls this repo old and deprecated, left up for posterity. That note is a fair summary of where the ecosystem has moved, but it does not diminish why this codebase became the standard first stop for people learning how language models actually train: every line you read is a line that runs.

Overview of nanoGPT: three entry scripts share one 330-line model, configurator files override globals at startup, and two preparation scripts feed uint16 token bins into the loop.

Reading the overview from left to right:

• train.py is the hexagon at the center: a while-True loop handling DDP setup, mixed precision, gradient accumulation, cosine learning-rate decay, evaluation, and checkpointing.
• sample.py loads either your checkpoint or an OpenAI GPT-2 variant and prints generated text with temperature and top-k controls.
• bench.py is train.py stripped to its skeleton, burning in for ten steps then timing twenty, optionally under the PyTorch profiler.
• model.py is the GPT itself, from LayerNorm through the Block to the GPT container, weight tying, scaled initialization, and GPT-2 weight import.
• configurator.py is the self-described “Poor Man’s Configurator”: it execs a config file into globals, then applies –key=value overrides with strict type checks.
• train_shakespeare_char.py, finetune_shakespeare.py, and train_gpt2.py are plain Python files of settings, one per scenario.
• data/shakespeare_char/prepare.py downloads tiny Shakespeare, builds a 65-character vocabulary, and writes train.bin, val.bin, and meta.pkl in seconds.
• data/openwebtext/prepare.py tokenizes about 8 million OpenWebText documents into roughly 9 billion tokens stored as a 17GB uint16 memmap.

Why You Need This

If you have ever read a transformer paper and then opened a production training repository, you know the gap. Industrial codebases route the forward pass through engine classes, configuration registries, distributed wrappers, and scheduler objects, and the actual mathematics is scattered across all of them. nanoGPT closes that gap completely. In model.py you can trace one token from the wte embedding table through twelve pre-norm blocks to the lm_head projection without ever leaving the file, and in train.py you can watch the gradient of that token get clipped, scaled, and applied in under a hundred lines. For anyone teaching or learning the mechanics, this is the reference implementation that other references cite.

The second reason is that the repo is a working GPT-2 laboratory, not a pedagogical stub. from_pretrained in model.py pulls real OpenAI checkpoints (gpt2, gpt2-medium, gpt2-large, gpt2-xl, at 124M, 350M, 774M, and 1558M parameters) from HuggingFace, transposes the Conv1D weights that GPT-2’s original TensorFlow layout requires, and even performs surgery to shrink the context window afterward. The README’s baseline table evaluates all four against OpenWebText, and config/finetune_shakespeare.py initializes from gpt2-xl at a constant 3e-5 learning rate to adapt the largest checkpoint to Shakespeare in about twenty iterations. Studying how little code stands between “download 1558M pretrained parameters” and “finetune them on your text” is genuinely instructive.

The third reason is honesty about performance. model.py ships an estimate_mfu method that computes model flops utilization analytically, counting 6N plus 12LHQT flops per token against the 312 TFLOPS bf16 peak of an A100, and train.py smooths that measurement with an exponential moving average printed every log interval. bench.py exists purely to measure iteration time and MFU without training overhead. Very few small repositories teach you to measure training efficiency rather than assume it, and that habit matters more the moment you move to real hardware budgets.

The detail view: GPT core wiring on the left, the pre-norm block with its flash and manual attention paths, the training loop's precision and checkpoint machinery, and the data plus config sources feeding both.

The model internals reward a slow read. LayerNorm is redefined by hand solely because PyTorch’s built-in norm cannot take bias=False. CausalSelfAttention projects one Linear to 3 times n_embd so query, key, and value for every head arrive in a single batched matmul; if torch.nn.functional.scaled_dot_product_attention exists (PyTorch 2.0 or newer) it takes the fused causal path, otherwise a registered tril mask buffer handles causality manually. GPTConfig pads the GPT-2 vocabulary from 50257 to 50304 because a multiple of 64 keeps tensor cores happy. The token embedding and the output head share one weight matrix through tying, the residual projections get the GPT-2 paper’s scaled initialization of 0.02 divided by the square root of twice the layer count, and at inference the lm_head runs only on the very last position, a small but real decoding optimization.

train.py makes the same economy look effortless. It reads RANK, LOCAL_RANK, and WORLD_SIZE from the torchrun environment, divides gradient accumulation steps by world size, and offsets each rank’s seed so data ordering differs. The data loader is deliberately primitive: get_batch recreates a numpy memmap every call to sidestep a known memory-leak pattern, samples random windows, and ships pinned tensors to the GPU with non_blocking copies while the previous forward runs. Precision is a two-line autocast context, with fp16 automatically enabling a GradScaler. Gradient accumulation syncs gradients only on the final micro step by toggling require_backward_grad_sync directly, an explicit rejection of the no_sync context manager that the comment calls bloated. torch.compile wraps the model in one line, which the README credits with cutting iteration time from roughly 250ms to 135ms when it landed. The learning-rate schedule is linear warmup for 2000 iterations then cosine decay toward min_lr, and checkpoints bundle the model, optimizer state, model_args, iteration, and best loss into one ckpt.pt.

Resume logic shows the same respect for reality. When init_from is set to resume, the script reloads the six model architecture fields from the checkpoint (forcing consistency where it matters, while letting dropout and other knobs change), then strips the _orig_mod. prefix that torch.compile sometimes bakes into saved state dict keys, a wart the comment admits is not fully understood. always_save_checkpoint can be flipped off so only improving validation losses hit the disk, which is exactly what the Shakespeare character config does since that run is expected to overfit. An eval_only flag exits after the first evaluation, which is how the GPT-2 baseline table in the README was produced without any training at all, and optional Weights and Biases logging ships configured but disabled by default.

The supporting cast is just as deliberate. configurator.py execs a config file into the caller’s globals, then parses –key=value arguments with literal_eval and asserts the replacement type matches the default, which catches most typos in about forty lines. The character-level Shakespeare preparation script encodes 1,115,394 characters into a 65-token vocabulary, splits 90/10 into 1,003,854 training and 111,540 validation tokens, and saves the stoi and itos mappings in meta.pkl so sample.py can decode generations without tiktoken. The OpenWebText preparation script instead uses the GPT-2 BPE encoder, appends the end-of-text token after each of 8,013,769 documents, holds out 4,007 documents for validation, and streams the result into memmap bins across 1,024 shards. Nothing is abstracted that does not need to be.

Even the side materials carry weight. The repo includes two notebooks: one works through transformer parameter counting and the arithmetic of where 124M parameters actually live, the other explores neural scaling laws by fitting curves to published runs. The README points readers to the Zero To Hero video series for the built-up-from-scratch version of the same material, runs its experiments on Lambda Labs GPUs, and hosts discussion in a Discord channel. bench.py deserves a mention too: beyond the simple burn-in-then-time mode, a profile flag wraps the same loop in the PyTorch profiler with a five-step wait, warmup, and active schedule, exporting TensorBoard traces with flop counting enabled, so the same file serves both quick MFU numbers and kernel-level inspection.

From Install to Shakespeare in Three Minutes

The README’s quick start is the fastest legitimate path from pip install to watching a transformer write pseudo-Shakespeare:

pip install torch numpy transformers datasets tiktoken wandb tqdm
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.py
python sample.py --out_dir=out-shakespeare-char

That config trains a baby GPT (6 layers, 6 heads, 384 embedding channels, 256-character context, batch 64, dropout 0.2) for 5000 iterations, reaching a best validation loss of about 1.4697 in roughly three minutes on one A100. Without a GPU, the README’s CPU recipe shrinks the model to 4 layers, 4 heads, and 128 channels with a 64-character context, finishing in a similar time at a loss near 1.88; on Apple Silicon, –device=mps engages the on-chip GPU for a two-to-three times speedup. At the serious end, torchrun –standalone –nproc_per_node=8 train.py config/train_gpt2.py launches the full 124M reproduction with about 0.5 million tokens per iteration toward 300 billion total, and eval configs reproduce the GPT-2 family baselines from 3.12 validation loss at 124M down to 2.54 at 1558M.

Honest limits: the repo trains only what train.py hardcodes, dense next-token prediction with DDP, so there is no FSDP, no tensor or sequence parallelism, no evaluation harness beyond held-out loss, and the README’s own todo list still mentions rotary or alibi embeddings and zero-shot benchmarks as unexplored. torch.compile is assumed on by default and remains unsupported on some platforms, Windows included, so –compile=False is the documented escape hatch. And the repository itself is now explicitly frozen in favor of nanochat, so treat it as a textbook edition: the cleanest 330 lines of GPT you will ever read, frozen at the exact moment it taught a generation how the full training loop fits on one screen.

Enjoyed this post? Never miss out on future posts by following us