Most multimodal papers describe their models with a diagram and leave the wiring to your imagination. BLIP — Bootstrapping Language-Image Pre-training, released by Salesforce AI Research in 2022 — is nicer than average: the whole system lives in a handful of readable PyTorch files that map almost one-to-one onto the paper’s losses. One ViT encodes images, one modified BERT handles text in two modes, and three losses (contrastive, matching, and language modeling) flow through those same two towers. Because the learned weights transfer to captioning, retrieval, VQA, and NLVR2, the repo is a compact case study in how a shared multimodal backbone gets specialized.
This post tours the salesforce/BLIP repository as it stands on main. We read the vision encoder, the cross-attention text model called MED, the pretraining loop that sums the three losses into one backward pass, and the thin task heads that wrap the backbone for each benchmark. Where a function matters, we link the exact line. Two notes first: the README opens with a deprecation banner — Salesforce now points people at LAVIS and later models — and the pins are frozen in early-2022 versions (transformers==4.15.0, timm==0.4.12, fairscale==0.4.4). Neither makes the code less instructive; both matter if you plan to run it today.
If you know clip, the contrastive image-text model that popularized the area, BLIP feels familiar: it inherits the dual-encoder idea and the big in-batch similarity matrix, then adds generation on top. The bert lineage is equally visible — the text tower is a vendored copy of Hugging Face’s BERT with surgical modifications.
The overview graph: a ViT vision tower and a MED text tower feed one pretraining model, which is then wrapped by four task heads. Every node links to a real file in the repository.
Reading the overview from left to right:
- VisionTransformer, L113 — the vision tower: a timm-derived ViT turning a 384-pixel image into 577 patch embeddings of width 768 (ViT-B) or 1024 (ViT-L).
- Block, L89 — one pre-norm transformer block; the last few can be wrapped with fairscale activation checkpointing when GPU memory runs short.
- interpolate_pos_embed, L281 — bicubic resizing of the learned position grid, the trick that lets a 224-pixel checkpoint drive a 384-pixel model.
- BertLayer, L320 — the modified BERT layer that only runs its cross-attention sublayer in
multimodalmode. - BertSelfAttention, L97 — the cross-attention variant projects keys and values from the vision width instead of the text hidden size.
- BLIP_Pretrain, L19 — the pretraining assembly: both towers, momentum copies, a 57,600-entry feature queue, and a forward pass returning three losses.
- BLIP_Decoder, L78 — the captioning specialization pairing the ViT with a causal BERT decoder.
- BLIP_ITM, L10 — the image-text matching head with two projections and a two-way classification head.
- train(), L30 — the loop that adds the three losses and steps AdamW, logging through MetricLogger, L92.
The Three Losses That Share One ViT
The heart of the repo is the forward pass of BLIP_Pretrain in models/blip_pretrain.py. It returns loss_ita, loss_itm, loss_lm at L212, and the loop in pretrain.py simply adds them at L61 before one backward(). Three objectives, one graph — that is the whole pretraining recipe.
The first loss, named loss_ita in code (the paper calls it ITC), is pure clip-style contrastive learning. Image features come from the CLS token of the ViT through a vision_proj linear layer; text features come from the CLS of the BERT encoder run in unimodal mode='text' through text_proj. Both are L2-normalized, multiplied into a similarity matrix, and divided by a learnable temperature clamped between 0.001 and 0.5. The interesting part is how the targets are built: momentum encoders — updated by _momentum_update, L225 with an EMA coefficient of 0.995 — produce soft targets for the current batch, and a feature queue maintained by _dequeue_and_enqueue, L232 extends the effective batch to 57,600 pairs without extra GPU memory. Because COCO captions are not unique — five captions can describe one image — the retrieval variant passes dataset indices and converts duplicate hits into soft positives rather than false negatives.
The second loss, loss_itm, reuses the same BERT encoder in multimodal mode: the caption’s tokens attend to the image’s patch embeddings through cross-attention, and a linear itm_head classifies the fused CLS token as match or mismatch. The negative pairs are not random — the code samples them from the ITC similarity distribution itself (L154-L173), choosing, for each text, the most confusable wrong image. This hard-negative mining is why the matching head ends up sharp enough to double as a filter later.
The third loss, loss_lm, trains generation. The caption tokens are fed to a causal BertLMHeadModel conditioned on image embeddings, with padding masked to -100 so only real tokens contribute. One subtle pretraining detail: the decoder is initialized by copying the encoder weights with tie_encoder_decoder_weights, L270, so the language model starts from the same text distribution the encoder sees. On top sits the alpha ramp — the loop computes alpha = config['alpha']*min(1,(epoch*len(data_loader)+i)/(2*len(data_loader))) at L58, slowly blending the momentum-model soft targets into the loss, the CapFilt-style bootstrapping trick of the paper.
MED: The Text Model With a Switch
models/med.py is the file people mean when they say “BLIP’s text model.” MED stands for Mixture of Encoder-Decoder, and the mixture is implemented with almost no new architecture: it is a vendored copy of Hugging Face BERT (the header cites transformers v4.15.0’s models/bert directory) plus two small changes.
Change one: every BertLayer can own a second attention module. At L328 the layer builds self.crossattention when the config requests it, and its forward method checks the mode argument — only when mode=='multimodal' at L358 do the cross-attention sublayers run. Change two: in the cross-attention BertSelfAttention, the key and value projections are sized from config.encoder_width (L112-L114) — the vision tower’s output width — rather than the text hidden size. That single line is the seam between the two towers: image patches enter the text stack as keys and values, exactly as the overview diagram shows.
The mode flag threads through BertEncoder down to every layer, so one BertModel serves three roles: unimodal text encoder for contrastive features, multimodal encoder for matching and VQA questions, and backbone for the causal decoder. The decoder half, BertLMHeadModel, L811, adds the label-smoothed next-token loss at L914 and the Hugging Face generation plumbing (prepare_inputs_for_generation, _reorder_cache for beam search), which the captioning head later calls with num_beams=3.
For captioning, models/blip.py wraps that decoder in BLIP_Decoder, L78 with a short prompt — "a picture of " — whose tokens are masked out of the loss. Its generate method (L128) repeats the image embeddings per beam, seeds the decoder with the prompt, and either beam-searches or nucleus-samples; the branch at L156 is the deterministic beam path.
From Install to Your First Model
The repository was tested on PyTorch 1.10, and its requirements.txt pins exactly four entries: timm==0.4.12, transformers==4.15.0, fairscale==0.4.4, and pycocoevalcap. Newer transformers releases move internal APIs around, so the pins are load-bearing:
pip install timm==0.4.12 transformers==4.15.0 fairscale==0.4.4
Next, grab the COCO-finetuned captioning checkpoint from the README’s finetuned-checkpoints table: model_base_caption_capfilt_large.pth from the sfr-vision-language-research bucket. Then the captioning path from the official Colab notebook is four lines of real code:
import torch
from PIL import Image
from torchvision import transforms
from torchvision.transforms.functional import InterpolationMode
from models.blip import blip_decoder
model_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_caption_capfilt_large.pth'
model = blip_decoder(pretrained=model_url, image_size=384, vit='base')
raw_image = Image.open('demo.jpg').convert('RGB')
transform = transforms.Compose([
transforms.Resize((384, 384), interpolation=InterpolationMode.BICUBIC),
transforms.ToTensor(),
transforms.Normalize((0.48145466, 0.4578275, 0.40821073),
(0.26862954, 0.26130258, 0.27577711)),
])
image = transform(raw_image).unsqueeze(0)
with torch.no_grad():
caption = model.generate(image, sample=False, num_beams=3,
max_length=20, min_length=5)
print(caption[0])
The constructor blip_decoder, L172 builds the ViT and decoder, then load_checkpoint fetches the weights — from URL or local path — and asserts no keys went missing. The normalization constants are the standard CLIP statistics. For the Hugging Face port, see our transformers tour; it wraps the same weights behind BlipProcessor/BlipForConditionalGeneration without the pinned-dependency ceremony.
Checkpoint Loading and the 384-Pixel Problem
Almost every checkpoint load hinges on one detail. The released ViT weights were pretrained at 224 pixels — a 14×14 patch grid plus a CLS token, 225 position rows — while captioning runs at 384 pixels, a 24×24 grid with 577 rows. Reusing the weights means the position embedding matrix must grow.
That happens in load_checkpoint, L215: after torch.load, it calls interpolate_pos_embed for the student ViT (and the momentum ViT during pretraining) at L226. The helper, interpolate_pos_embed, L281, splits the checkpoint rows into extra tokens (CLS and any special tokens) and the patch grid, reshapes the grid to 2-D, runs bicubic torch.nn.functional.interpolate at L297 to the new resolution, and concatenates the untouched token rows back on front. Keys whose shapes still disagree are silently deleted (L232-L233) before a strict=False load, which is how heads like itm_head can be freshly initialized on top of a pretrained base.
The vision tower itself is deliberately unexciting: models/vit.py states in its header that it is based on the timm code base, and create_vit at L194 configures either 12 blocks of width 768 or 24 blocks of width 1024. The only BLIP-specific additions are hooks on Attention that save attention maps for visualization, and the fairscale checkpoint_wrapper toggle for gradient checkpointing on the deepest blocks.
Task Heads: Matching, VQA, Retrieval
Each downstream model is a thin shell around the same two towers. BLIP_ITM, L10 is the smallest: its forward takes a match_head argument — 'itm' returns the two-way classifier score over the fused CLS token, while 'itc' returns the raw cosine similarity between the projected image and text features.
BLIP_VQA, L10 is the most inventive. Question answering is cast as generation — the question is encoded multimodally, and a decoder emits a short answer — but at inference you can pick 'generate' (beam search with num_beams=3, max_length=10) or 'rank'. The rank path, rank_answer, L120, scores the candidate set: it reads first-token logits, keeps the top-k answers by first-token probability, then sums each full answer’s log-probability under the decoder and returns the argmax.
BLIP_Retrieval, L10 is pretraining minus the LM loss, plus two touches: dataset indices turn duplicate captions into soft positive targets, and a negative_all_rank flag mines hard negatives across all distributed workers, gathering image embeddings gradient-preserving through a custom autograd function so the ITM loss stays differentiable across ranks.
The detail graph adds the internals: ViT block composition, the MED cross-attention seam, checkpoint loading, the momentum queue, and the pretraining loop's calls into utils.
Honest Limits
Read the first line of the README: the repository is deprecated, and Salesforce explicitly recommends not using it for production. It survives as a reference implementation, integrated into LAVIS and superseded architecturally by BLIP-2 (which freezes the image encoder and an LLM instead of pretraining a bespoke text tower) and InstructBLIP. The scale is modest by current standards — a ViT-B/L backbone at 384-pixel resolution, at most 129 million pretraining images — and caption quality has moved on considerably since.
The dependency pins are the practical obstacle in 2026: Enjoyed this post? Never miss out on future posts by following us transformers 4.15-era code touches internal module paths that no longer exist in modern releases. If your goal is a working captioning or VQA system, use the Hugging Face port (transformers); if your goal is understanding how a multimodal backbone is assembled and bootstrapped, this codebase still has few peers. For the contrastive-only ancestor whose similarity matrix BLIP repurposed for hard-negative mining, read our clip tour; for the surrounding landscape of inference runtimes and training frameworks, start from the inference overview. 
