When OpenAI released CLIP in 2021 it changed how the community thinks about vision-language pretraining: two towers, a temperature-scaled contrastive objective, and zero-shot classification from prompt templates. The original repository — which we walked through in the earlier tour of /clip/ — shipped the weights and a minimal inference API, but training was never its focus. Its code froze shortly after release, and anyone who wanted to retrain, fine-tune, or extend CLIP had to start over. That gap is exactly what mlfoundations/open_clip fills. Begun as a community reimplementation of the contrastive objective, it has grown into the de facto maintained implementation: hundreds of pretrained checkpoints, native support for the SigLIP, CoCa, CLAP and MaMMUT model families, timm and HuggingFace backbones behind one factory function, and a full training package that scales from a single GPU to a multi-node cluster.

The library is deliberately thin. It is pure Python on top of PyTorch — no compiled extensions, no Cython to build, nothing to fight with on Windows or ARM — and it leans on two ecosystem workhorses that most projects already trust: timm for vision backbones and transformers for text encoders, the same library behind /transformers/. That combination is the quiet design decision that makes open_clip feel bigger than it is. A ViT-B-32 config is a handful of JSON keys; a ConvNeXt or EVA vision tower is borrowed from timm at instantiation time; a multilingual XLM-R text tower is borrowed from transformers the same way. The library adds contrastive glue, checkpoint conversion, and a registry, not another modeling framework.

This post is a source tour, in the same style as our previous posts. We start at the function every user touches first — the model factory — then descend into the pretrained-weight registry, the tokenizer bridge, and the distributed training and zero-shot evaluation flow. Two dependency graphs drawn from the actual repository structure anchor the narrative: an overview of the whole system and a detail view of how the pieces call each other.

Overview of open_clip: the training entry wires the factory, the weight registry, the tokenizer bridge, the dual-tower models, and the train/eval flow into one call graph.

Reading the overview from left to right:

• main, L62 — the CLI entry that parses arguments, initializes the distributed device, creates the model and data, and drives the epoch loop.
• create_model_and_transforms, L1251 — the one-stop factory returning a model plus paired training and validation preprocess transforms.
• _PRETRAINED, L460 — the curated weight registry mapping every model and tag to a checkpoint URL or HuggingFace repo.
• CLIP, L412 — the canonical dual-tower model with encode_image / encode_text and a learnable logit_scale.
• VisionTransformer, L550 — the native ViT image tower; the same file hosts the causal-masked text tower and attention blocks.
• HFTokenizer, L625 — the HuggingFace AutoTokenizer bridge that adapts arbitrary tokenizers to CLIP conventions.
• TimmModel, L25 — the adapter that turns any timm classification backbone into a vision tower.
• train_one_epoch, L296 — the autocast step loop with gradient accumulation, loss reduction, and metric logging.
• build_zero_shot_classifier, L22 — builds class embeddings from prompt templates for end-of-epoch evaluation.

The Factory: One Entry Point, Many Model Families

Everything in open_clip funnels through create_model_and_transforms in src/open_clip/factory.py. It delegates to create_model, L326, which first parses the model identifier. A bare name like ViT-B-32 is looked up in the built-in configs under src/open_clip/model_configs/ — roughly 150 JSON files, each describing embed_dim, vision_cfg, and text_cfg. A prefixed name like hf-hub:org/repo instead fetches an open_clip_config.json from the HuggingFace Hub, and local-dir: reads the same format from disk. Three input styles, one config schema.

Class selection is the interesting part. Rather than hard-coding class names, the factory calls traits_from_config, L144 to classify the config into a ModelFamily, then picks the implementation: plain CLIP for the classic flat layout, CustomTextCLIP (L613) when the text tower should stay a nested module — required for HF-backed, “modern”, or variable-length text towers — plus CoCa, CLAP, MaMMUT, and the generative NaFlex classes. Once instantiated, the model exposes the familiar encode_image, L488 and encode_text, L508 methods, both with an optional normalize flag.

Weight loading is its own sub-flow. load_checkpoint, L255 accepts .safetensors files through the safetensors reader and everything else through torch.load with weights_only=True by default. Before the state dict reaches the model it is massaged: third-party checkpoints are remapped by convert_state_dict, L278, legacy flat-layout checkpoints are re-keyed into the custom-text format, logit-scale shapes are reconciled, and vision or text position embeddings are interpolated if the checkpoint’s grid size differs from the requested one. This is where years of ecosystem drift get absorbed so that a 2021 OpenAI checkpoint and a 2026 community checkpoint both load through the same call.

The Pretrained-Weight Registry

The registry in src/open_clip/pretrained.py is what makes tags like openai, laion2b_s34b_b79k, or datacomp_xl_s13b_b90k mean anything. _PRETRAINED, L460 is a large dictionary keyed by architecture, where each value maps tags to config dicts built by helpers such as _pcfg for OpenAI-style defaults. Crucially, the config carries more than a URL: it records the normalization mean/std, the resize mode, the interpolation, and a quick_gelu flag, because the original OpenAI models were trained with the QuickGELU activation and silently degrade if you run them with vanilla GELU. When a tag’s activation setting disagrees with the requested architecture variant, the factory emits a QuickGELU mismatch warning rather than letting accuracy evaporate quietly.

Lookup and download are separated cleanly. get_pretrained_cfg, L806 answers “what are the settings for this tag”, while download_pretrained, L922 resolves the checkpoint to a cached file, preferring GitHub release URLs or HuggingFace Hub downloads (download_pretrained_from_hf at L885). Over the project’s life most weights have been mirrored from GitHub releases to timm/* and laion/* Hub repos, which matters because release assets lack CDN-grade bandwidth. Several newer checkpoints ship as safetensors — the format we covered in /safetensors/ — which the loader detects by file extension and reads without pickle deserialization. For pure inference there is a shortcut, create_model_from_pretrained, that fuses model creation and transform construction in one call.

Tokenizers: the HuggingFace Bridge

The original CLIP used a bespoke byte-pair encoder trained on its caption corpus. open_clip keeps that exact implementation as SimpleTokenizer, L264, complete with the bundled BPE vocabulary, so OpenAI checkpoints tokenize identically here and in the original library. Everything newer, though, rides on HFTokenizer, L625, a wrapper around AutoTokenizer — the same dispatch layer we described in /tokenizers/. The wrapper does three things that look small and are not: it forces right-side padding so CLIP’s eos-pooling math stays correct for decoder-family tokenizers that default to left padding, it maps bos/eos/cls/sep ids into the sot/eot vocabulary the towers expect, and it applies a configurable text-cleaning chain before encoding.

Selection happens in get_tokenizer, L850: the model’s text_cfg decides between the HF bridge, the SigLIP sentencepiece tokenizer (SigLipTokenizer, L896 with Gemma, C4, or multilingual variants), a tiktoken-based tokenizer for the generative models, or the original BPE. The same text_cfg also declares context_length — 77 for classic CLIP, much longer for SigLIP and modern variants — and the factory cross-checks pad and eos ids against the pooling mode so a generative model cannot silently train with the wrong ignore index. The lesson from reading this file is that “just tokenize it” hides most of the compatibility surface in multimodal systems.

From Install to Your First Model

The package installs from PyPI under its distribution name open_clip_torch:

pip install open_clip_torch
# training extras (webdataset, pandas, transformers):
pip install "open_clip_torch[training]"

Then the full factory round-trip in a few lines:

import torch
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k"
)
tokenizer = open_clip.get_tokenizer("ViT-B-32")

image = preprocess(Image.open("dog.jpg")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])

with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(text_probs)  # per-class probabilities

Note what you did not do: no manual config parsing, no weight conversion, no activation-function bookkeeping — the tag carried all of it. Every API used above (create_model_and_transforms, get_tokenizer, encode_image, encode_text, the laion2b_s34b_b79k tag) exists verbatim in the current source tree, and the returned preprocess object is built by image_transform_v2, L501 from torchvision primitives on the mean/std the registry recorded for that tag.

Training, Distributed by Default

The open_clip_train package is where “reproduction” stops being a claim and becomes a script. main, L62 enables tf32, calls init_distributed_device, L80 — which reads torchrun’s LOCAL_RANK/RANK/WORLD_SIZE environment variables or SLURM’s equivalents, initializes torch.distributed with nccl, gloo, or hccl, and pins the device to the local rank — then assembles data with get_data, L1203, the optimizer with create_optimizer, L336 (weight-decay exclusions for norms, biases, and the logit scale, plus optional layer-wise decay), and the schedule with cosine_lr, L74. Precision is centralized in get_autocast, L6: one flag selects amp fp16, pure fp16, bf16, or fp32, and the loop picks the matching input dtype and gradient scaler.

The step loop in train_one_epoch is honest engineering rather than magic. Each batch flows through the task object — the TrainingTask contract (L30) with CLIPTask (L9) as the contrastive implementation — whose forward returns a dict of losses computed by ClipLoss, L57, a symmetric cross-entropy over the scaled image-text similarity matrix. The SigLIP objective swaps it for the pairwise sigmoid SigLipLoss, L375, and distillation and CoCa variants subclass the same base. Because open_clip is pure PyTorch underneath, every collective here is a plain dist.all_reduce — the native code path we touched on in /pytorch/ — so nothing exotic hides inside the C extensions; there are none. An --local-loss option slices the contrastive matrix per rank to save memory, and evaluation runs after each epoch through the zero-shot classifiers built from OPENAI_IMAGENET_TEMPLATES, L2, the same 80 prompts the original paper used.

Detail view: CLI and IO, the factory and its weight plumbing, the model families, tokenizer and preprocess builders, the task abstraction, the training loop, and the zero-shot eval chain.

Honest Limits

No tour is complete without the trade-offs. First, pretrained-weight licensing varies dramatically across the registry: the OpenAI originals, LAION-2B, DataComp, SigLIP, and MetaCLIP checkpoints come with different licenses and different dataset provenance, and open_clip deliberately does not normalize that away — you are expected to check the tag’s source repo before deploying. Second, checkpoint size management is on you. Registry entries now span 150 MB RN50s to multi-gigabyte ViT-bigG and EVA models; the library caches whatever you download, offers no deduplication or sharding, and load_checkpoint will happily keep both the flat and custom-text copies of a state dict in memory during conversion. Plan disk and RAM accordingly.

Third, training-data dependence is structural. CLIP-family models inherit the curation, biases, and coverage of their caption corpora — a DataComp-tuned ViT-B-32 and a LAION-tuned one of the same architecture are not interchangeable, and zero-shot gaps on underrepresented classes trace directly to what the contrastive objective saw. Fourth, tokenizer differences matter more than users expect: the original CLIP BPE, HF tokenizers, sentencepiece SigLIP variants, and tiktoken-based generative tokenizers are not drop-in swappable, text embeddings are not portable across checkpoints with different vocabularies, and the classic 77-token context length is an architectural constraint, not a soft default. Finally, quickGELU and preprocessing mismatches are silent until you evaluate — the warnings exist, but only comparison against published numbers will catch a subtle preprocessing drift. Treat the registry’s mean/std/interpolation fields as part of the model definition, because that is how the source treats them.

Enjoyed this post? Never miss out on future posts by following us