When OpenAI released CLIP (Contrastive Language-Image Pre-training) in January 2021, the accompanying paper made a claim that sounded almost careless: train on 400 million (image, text) pairs scraped from the web, and the resulting model can name things it was never explicitly taught. The headline demo was zero-shot ImageNet classification — matching the performance of a ResNet-50 trained on all 1.28 million labeled examples, without ever seeing one of those labels. What is more interesting for readers of this series is the artifact itself: the released repository is tiny. One model file, one loader, one tokenizer, and a gzipped vocabulary. No training code, no data loaders for the 400-million-pair corpus, no experiment framework. Just the trained dual encoder and the minimum scaffolding needed to run it.
The idea is easier to state than to train. Two encoders run side by side: an image encoder (either a modified ResNet or a Vision Transformer) and a text encoder (a masked self-attention stack in the GPT-2 family, architecturally a cousin of the models covered in our /transformers/ tour). Each encoder projects into a shared embedding space of identical width. During training, a batch of N image-text pairs produces an N-by-N matrix of cosine similarities, and a symmetric cross-entropy loss pushes the N correct pairings toward 1 while suppressing the N-squared-minus-N wrong ones. At inference, any string can stand in for a class name, and classification becomes a similarity lookup. If you have followed our /nanogpt/ tour, the text side will feel immediately familiar: token embedding, learned positional embedding, causal multi-head attention, QuickGELU MLPs.
The public surface is deliberately narrow. clip.available_models(), clip.load(), clip.tokenize() — three functions, nine checkpoints, one preprocessing chain built from the same torchvision primitives we walked in our /torchvision/ tour, and a byte-pair-encoding tokenizer that anticipates the modern wave of tokenizer tooling we covered in our /tokenizers/ tour. This post walks the actual files on main with line-verified anchors, then closes with the trade-offs the authors wrote down.
Repository flow of openai/CLIP: the package API in clip/clip.py feeds build_model in clip/model.py, the dual-encoder definition; clip/simple_tokenizer.py loads the gzipped BPE vocabulary; setup.py, the CI workflow and the consistency test hold the packaging and verification perimeter.
Reading the overview from left to right:
load, L94 — the single entry point: resolves a model name or local checkpoint path, downloads with a SHA-256 gate, and returns(model, preprocess)._MODELS, L30 — a flat dict of nine checkpoint URLs (RN50 through ViT-L/14@336px); each URL embeds its own checksum as the second-to-last segment.build_model, L399 — reconstructs every hyperparameter from checkpoint tensor shapes, so no config file ever ships with a model.CLIP, L243 — the dual-encoder wrapper that owns both towers, the projections, and the learned logit scale.VisionTransformer, L206 — the image tower for ViT variants: patch-conv embedding, class token, transformer, and a learned projection to the joint space.Transformer, L195 — the shared residual-attention stack, instantiated once for text and again (maskless) inside the ViT.SimpleTokenizer, L62 — the byte-level BPE tokenizer whose entire vocabulary lives in one gzip file next to the code.tokenize, L205 — wraps raw text into start-of-text / end-of-text tokens and pads every sequence to the fixed 77-token context.test_consistency, L9 — the one test that matters: JIT and eager loads of every model must agree on softmax probabilities.
Why Contrastive Pretraining Is the Whole Trick
Everything CLIP does at inference is a consequence of how CLIP.forward, L358 is written. Both towers encode their inputs, each feature vector is L2-normalized along the batch dimension, and the logits are nothing more than a scaled matrix product: logit_scale * image_features @ text_features.t(). There is no classifier head anywhere. The logit scale, L295 is a learned scalar initialized to ln(1/0.07) ≈ 2.66 — an exponential temperature of about 14.3 — because raw cosine similarities in [-1, 1] are far too flat to drive a softmax over thousands of negatives.
That design is what makes zero-shot transfer work. To classify an image into K categories, you do not load new weights; you encode K caption templates with the same text tower used during training and pick the argmax row. The paper’s ablations showed the prompt wording matters enormously, which is why the repository ships data/prompts.md — the exact classes-and-templates used for 26 of the 27 zero-shot datasets in the paper, with the 80-template ImageNet ensemble relegated to a notebook.
Inside model.py: Two Encoders, One Joint Space
The smartest few lines in the repository are at the top of build_model, L399. The loader never carries an architecture config; instead the function interrogates the state dict. If "visual.proj" is present, the image tower is a ViT and the code reads width, patch size, and layer count straight out of tensor shapes — vision_patch_size from visual.conv1.weight.shape[-1], the grid from the positional-embedding length, and the transformer depth by counting distinct attn.in_proj_weight keys. Otherwise the tower is a ResNet and the counts come from the visual.layer{1..4} key groups. Every dimension, including the joint embed_dim and the 77-token context, is inferred rather than declared.
The ViT tower itself, VisionTransformer, L206, is a compact 35 lines. Patchification is a single nn.Conv2d whose kernel size equals its stride equals the patch size (conv1, L211) — for ViT-B/32 that chops a 224-pixel image into a 7-by-7 grid of 32-pixel patches. Its forward, L223 reshapes the grid into a token sequence, prepends the class embedding, adds learned positional embeddings, runs the stack, then takes only the class-token position, applies a post LayerNorm, and matmuls with the proj matrix into the joint space.
Two details in the building blocks reward attention. QuickGELU, L166 is x * sigmoid(1.702 * x) — not the exact GELU, but close enough at speed, and frozen into the checkpoints now, so any reimplementation must match it bit-for-bit to reproduce logits (our consistency test below enforces exactly that). And LayerNorm, L157 is a subclass that upcasts to float32 and back, a pragmatic fix for fp16 instability that the weight-conversion helper, L375 relies on when it halves all conv, linear, and attention weights at load time.
The ResNet branch is equally opinionated. ModifiedResNet, L94 replaces the usual 7x7 stride-2 stem with three 3x3 convolutions and swaps every strided convolution for an average-pool-then-convolve pair to reduce aliasing. Its head is not average pooling at all but AttentionPool2d, L58, a single-query multi-head attention over all spatial positions whose output projection lands directly in the joint embedding width.
The text tower reuses the same Transformer, L195 class, this time with a causal mask built by build_attention_mask, L328 — a plain upper-triangular tensor filled with negative infinity. encode_text, L343 then does something charmingly low-tech to pool the sequence: since the end-of-text token has the highest vocabulary id, text.argmax(dim=-1) locates the true end of every (padded) caption, and that position’s hidden state is projected into the joint space.
From Install to Your First Model
The README’s install line still works verbatim:
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.git
setup.py is a bare-bones packaging of the clip package, with include_package_data=True plus a one-line MANIFEST.in ensuring the 1.3 MB vocabulary gzip rides along in the wheel. The minimal round trip uses only APIs verified in the checked-out source:
import torch, clip
from PIL import Image
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
image = preprocess(Image.open("pic.jpg")).unsqueeze(0).to(device)
text = clip.tokenize(["a photo of a cat", "a photo of a dog"]).to(device)
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
logits_per_image, logits_per_text = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
print(probs)
Under the hood, load, L94 downloads the checkpoint into ~/.cache/clip, trusting the SHA-256 embedded in the URL — _download, L43 re-downloads with a warning if the local file’s hash drifts. It first tries torch.jit.load; on failure it falls back to a state dict and rebuilds the eager module via build_model. On CPU it forces float32. The returned preprocess comes from _transform, L79: resize and center-crop to the model’s native resolution with bicubic interpolation, convert to RGB, and normalize with the exact web-crawl statistics baked into the training run — (0.48145466, 0.4578275, 0.40821073) mean and (0.26862954, 0.26130258, 0.27577711) std, numbers you will meet in dozens of downstream repos. tokenize, L205 returns a padded int tensor of shape [batch, 77] and raises rather than silently truncating unless you pass truncate=True.
The Tokenizer: One Gzip File Is the Whole Vocabulary
clip/simple_tokenizer.py is 132 lines, and its universe of symbols is bpe_simple_vocab_16e6.txt.gz — a vocabulary learned over roughly 16.6 million text pairs. SimpleTokenizer.__init__, L62 unpacks the gzip, skips the first version line, and keeps the merge table slice merges[1:49152-256-2+1] — 48,894 learned merges. Add the 256 reversible byte characters from bytes_to_unicode, L16 (the GPT-2 trick of remapping raw bytes onto printable unicode so BPE never emits UNKs), the same 256 with an end-of-word marker, and two special tokens, and the encoder lands at exactly 49,408 entries — the vocab_size every checkpoint reports.
Encoding is equally transparent. encode, L121 runs basic_clean, L50 (ftfy plus a double HTML unescape), then whitespace_clean, L56, lowercases, and splits with a case-insensitive regex that keeps contractions, letter runs, single digits, and punctuation runs apart. Each fragment is byte-mapped and greedily merged by bpe, L80, which repeatedly applies the lowest-ranked adjacent pair and caches results per token. decode, L129 inverts the mapping and turns end-of-word markers back into spaces. That is the contrast with industrial tokenizers: no configurability, no non-Latin handling — just the frozen artifact the model was trained against.
Zero-Shot Classification, Verified by One Test
Because the CI workflow pins Python 3.8 against PyTorch 1.7.1, 1.9.1 and 1.10.1, the repository’s only test has to be robust across three generations of the framework. It parametrizes over all nine model names, loads each checkpoint twice — once with jit=True, once eager — runs the CLIP.png figure through both, and asserts the softmax probabilities agree within atol=0.01, rtol=0.1. That is precisely the failure mode QuickGELU and the LayerNorm cast would cause if a reimplementation drifted.
For an end-to-end zero-shot loop, the README’s CIFAR-100 example remains the canonical template: encode one image, encode one caption per class (optionally averaging over prompt templates), softmax, sort. The Interacting with CLIP notebook walks the same path interactively, including probability inspection per prompt. Everything you might bolt on afterwards — embedding caches, image search, or the ONNX export we covered in our /onnx/ tour — consumes exactly the two encode_* methods.
Honest Limits
The repository is a frozen research artifact, and the authors say so in model-card.md: the models are “intended as a research output,” any deployed use is out of scope, and the training code and 400-million-pair dataset were never released. You get checkpoints and inference scaffolding, nothing more. The code also shows its age — it warns below PyTorch 1.7.1, and the JIT path reaches into torch-internal graph nodes to patch device constants.
The tokenizer is English-only by construction: the 49,408-entry BPE table was learned from English web text, and the model card restricts use to English prompts — multilingual successors like SigLIP had to rebuild both towers and vocabularies from scratch. Prompt phrasing is a real sensitivity, not a footnote: the paper’s own Table 9 ablations show large swings from template choices, which is why the exact templates ship as data. Resolution is capped by the checkpoint — Enjoyed this post? Never miss out on future posts by following us input_resolution is read from the state dict, 224 pixels for the standard ViTs (a 7x7 patch grid at patch size 32) with 336 available only as the separate ViT-L/14@336px checkpoint — so fine detail below the patch size is simply unreachable. And the 77-token context is a hard ceiling: tokenize either raises or silently replaces the tail of long captions with the end-of-text token, and since pooling reads exactly that end-of-text position, anything truncated is information the text tower never sees. Finally, treat the similarity outputs as logits-to-be-tempered, not probabilities: the learned logit_scale was fit during contrastive training, so raw cosine thresholds will not transfer across models or domains. 
