In February 2019, OpenAI published a short paper with a long shadow: “Language Models are Unsupervised Multitask Learners.” The model it described, GPT-2, was trained on WebText, a large scrape of outbound Reddit links, and it demonstrated something uncanny for its time — that a single objective, predicting the next token, could produce coherent paragraphs, answer factual questions, and even translate across languages without ever being explicitly taught to. The architecture was deliberately unremarkable: a decoder-only transformer with 48 layers and 1.5 billion parameters in its largest configuration. The scaling was the message, and the openai/gpt-2 repository is the receipt.

What makes this repository special is not the model size but the surface area. The entire neural network lives in one file, src/model.py, in roughly 170 readable lines. The generation loop fits in about 95 lines of src/sample.py. The tokenizer is a single self-contained class in src/encoder.py. There are no config frameworks, no dependency injection, no plugin registries — just five Python files in src/ and a download script. If you have ever wanted to see exactly what “scaled next-token prediction” meant in code, without abstraction layers yelling over each other, this is the cleanest artifact you will find from that era.

Why read it in 2026, when the state of the art has moved several generations ahead? Because GPT-2 is the direct ancestor of everything you currently use: the tokenizer you load from Hugging Face, the attention shapes in your favorite inference engine, the top-k and top-p knobs on your API calls. Modern successors like nanoGPT and llm.c re-derive this same architecture with better tooling, and libraries such as Transformers still expose GPT-2 checkpoints as first-class citizens. Understanding the original makes every derivative easier to read. Let’s walk through it.

The whole repository at a glance: two CLI entry points converge on a sampling loop that repeatedly calls a five-function transformer, with a byte-level BPE tokenizer on the side and a download script feeding everyone checkpoints.

Reading the overview from left to right:

• Get the weights first. download_model.py streams the checkpoint files — model.ckpt.data-00000-of-00001, encoder.json, hparams.json, vocab.bpe — from a public Azure blob store into models/<name>/. GPT-2 ships in four sizes: 124M, 355M, 774M, and 1.5B parameters.
• The interactive entry point. interact_model, L11 is the famous REPL. It builds the TensorFlow graph once, restores the checkpoint with tf.train.Saver, then loops on typed prompts: encode, run the session, decode, print.
• The unconditional twin. sample_model, L11 does the same but seeds the context with the <|endoftext|> token instead of user input, letting the model free-run forever. It is the closest thing this repo has to a demo of raw model personality.
• The tokenizer factory. get_encoder, L108 reads encoder.json (the 50,257-token vocabulary) and vocab.bpe (the merge table) and returns an Encoder object shared by both entry points.
• The generation loop. sample_sequence, L43 wraps a tf.while_loop that runs one forward step per token: fetch the logits, divide by temperature, filter with top-k and top-p, then draw a token with tf.multinomial.
• The model itself. model, L147 embeds token ids with wte and absolute positions with wpe, stacks n_layer transformer blocks, applies a final LayerNorm, and projects logits by tying weights back to the embedding matrix.
• Hyperparameters in one place. default_hparams, L5 hardcodes the 124M defaults — context 1024, embedding 768, 12 heads, 12 layers — and each larger model simply overrides these from its hparams.json.

The Tokenizer Under the Microscope

GPT-2’s tokenizer is the direct ancestor of the byte-level BPE used by GPT-3, GPT-4’s early tokenizers, and countless open models, so it deserves its own stop. The trick that made it famous is in bytes_to_unicode, L9: instead of treating raw UTF-8 bytes — which include whitespace, control characters, and multi-byte sequences — the encoder maps every possible byte value to a printable unicode character. This means the vocabulary never needs an <UNK> token; any string a user can type is reversible, byte for byte.

The Encoder class in encoder.py, L42 compiles a hand-tuned regex that splits text into contractions, letters, numbers, punctuation, and whitespace runs. Each chunk is byte-mapped, then bpe, L55 repeatedly merges the adjacent pair with the lowest rank from vocab.bpe until nothing more can merge, caching results so repeated words cost nothing. encode, L96 chains these steps into token ids, and decode runs the whole chain in reverse. Fifty-two lines of Python, and it still decodes modern GPT-2 checkpoints correctly. When people complain about GPT-2’s tokenizer mangling arithmetic or whitespace, this file is the crime scene.

Inside the Transformer Block

The detail view follows one call deep: entry scripts drive sample_sequence, which drives the model function, whose block layer fans out into attention, MLP, and LayerNorm primitives, while the tokenizer branch shows how raw text becomes token ids.

The block, L123 function is a textbook pre-LayerNorm residual unit: normalize, attend, add; normalize, project, add. Two things are worth noticing. First, the LayerNorm comes before each sublayer, not after — GPT-2 was one of the influential adopters of this ordering, which stabilized training at depth. Second, the norm is hand-rolled in norm, L28 as mean and variance reduction plus a learned affine pair g and b, rather than a framework call, because in TF1 era code you frequently controlled numerics yourself.

Attention lives in attn, L69. A single conv1d, L50 — which is really just a reshaped matmul, the TF1 equivalent of a dense layer — produces query, key, and value projections in one shot. The heads are carved out by split_states, L40, scores are masked by attention_mask, L58, whose docstring cheerfully explains it avoids tf.matrix_band_part because that op “produces garbage on TPUs,” and softmax weights the values. The function also manages the k/v cache: when past is provided, old keys and values are concatenated so each new token attends to everything before it without recomputation — the same incremental decoding trick every inference server still uses.

The MLP is refreshingly boring: mlp, L115 expands to four times the model width, applies the tanh-approximated gelu, L25, and projects back. Finally, positions_for, L141 computes absolute position ids offset by the cache length. There is no rotary embedding, no FlashAttention, no quantization — just the plain 2019 recipe, which is precisely why it reads so clearly.

One more detail in model.py rewards attention: weight tying. The output logits at the top of model, L147 are computed as tf.matmul(h_flat, wte, transpose_b=True) — the same embedding matrix that converts tokens to vectors is transposed to convert vectors back to token scores. No separate output projection, 50,257 times 768 parameters saved, and slightly better language modeling quality, a trick that traces back to Press and Wolf’s 2017 paper and survives in most decoder models today. The past handling is equally economical: the top-level function unstacks the cache along its second axis and hands each layer exactly its own slice, so block never needs to know where it sits in the stack. It is a small thing, but it shows how much expressive power you can extract from a plain for loop when the shapes are designed well.

From Install to Your First Model

The repository is pinned to the TensorFlow 1.x world, so set expectations before you start: requirements.txt asks for tensorflow==1.12.0 alongside fire, requests, tqdm, and regex. The smoothest path today is the provided Docker route — Dockerfile.cpu and Dockerfile.gpu wrap the old dependency stack so it does not fight your modern Python. Inside that environment, three commands get you text:

1. python download_model.py 124M — fetches the smallest checkpoint, about half a gigabyte.
2. python src/interactive_conditional_samples.py --model_name 124M --top_k 40 --length 256 — starts the REPL.
3. Type any prompt and press Enter; the model continues it.

The sampling flags map directly to functions you have already seen. top_k controls top_k_logits, L5, which masks every logit below the k-th largest with -1e10. top_p controls top_p_logits, L25, one of the earliest open implementations of nucleus sampling: sort the logits descending, keep the smallest prefix whose cumulative probability exceeds p, and mask the rest. Temperature divides the logits before filtering. The defaults are conservative — top_k=0 and top_p=1 mean no truncation at all — and the README’s suggestion of top_k=40 remains a sensible starting point for coherent output.

A note on scale before you get ambitious. The 124M checkpoint runs comfortably on a laptop CPU; the 355M still responds in seconds; the 774M and 1.5B variants want a real GPU and noticeably more RAM, since the TF1 graph materializes all weights in single precision with no quantization story whatsoever. Batch size only affects throughput and memory, not output quality, and nsamples must divide evenly by batch_size, an assertion the script checks up front. If you just want to watch the model ramble without a prompt, swap in generate_unconditional_samples.py — it never asks for input and prints an endless stream of samples separated by banner lines. Watching the same checkpoint produce condition-free text is a surprisingly good way to feel the difference between what the model “knows” and what your prompt contributes.

Honest Limits

This repository is an archived, unmaintained research artifact, and you should treat it that way. Nobody at OpenAI is fixing issues in it, the last meaningful updates date from 2019-2021, and the code targets APIs — tf.Session, tf.placeholder, tf.variable_scope, tf.train.AdamOptimizer-style graph building — that were removed from TensorFlow 2. Running the raw requirements.txt on a current machine is both painful and a genuine security concern: TensorFlow 1.12 has years of unpatched CVEs, and pulling it into a modern environment is not something to do casually. Use the Dockerfiles, use a throwaway virtual machine, or just read the code instead of executing it.

There are also scientific limits worth naming. The context window is a hard 1024 tokens, encoded as absolute position embeddings, so anything longer simply does not fit. The repo contains only inference code — the training setup described in the paper was never released here, so reproduction efforts like nanoGPT had to rebuild the whole loop from scratch. Efficiency-minded readers will find no batching scheduler, no continuous batching, nothing like the kernels in llama.cpp or llm.c. And if you want to trace how these ideas evolved into modern training stacks, tinygrad and Transformers make excellent next stops.

None of that diminishes the reason this repo endures. GPT-2 is the moment the field agreed that scale plus next-token prediction was a research direction worth betting on, and the code preserves that moment in amber: five files, one idea, zero ceremony. Read src/model.py once and you will recognize its fingerprints in every transformer you open afterward.

Enjoyed this post? Never miss out on future posts by following us