Every multimodal assistant that chats about images owes something to this repository. LLaVA — Large Language and Vision Assistant — is the research codebase behind visual instruction tuning, the NeurIPS 2023 oral that showed a frozen CLIP encoder, a frozen Vicuna language model, and one tiny trainable bridge between them could follow complex instructions about pictures. Its successor LLaVA-1.5 then reached state of the art on eleven benchmarks using only public data, training in about a day on a single eight-A100 node. The insight was deceptively simple: you do not need to co-train vision and language from scratch, you need to teach an LLM to read a grid of image features as if they were tokens.

The repository stays as small as the idea. Strip away evaluation scripts and demo servers, and the entire model is a few hundred lines: a constants file whose most important number is IMAGE_TOKEN_INDEX = -200, a mixin that splices projected image features into a token stream, a regex-parsed two-layer MLP, and a frozen CLIP tower. Everything else — the training recipe, the worker cluster, the chat templates — is scaffolding around that splice. It is one of the few famous papers whose full method you can verify by reading the source in an afternoon. As always in this series, what follows is an educational tour of published source code.

From the chat entry points through the controller-and-worker serving cluster, the checkpoint loader, and the vision-language bridge, over to the two-stage training script.

Reading the overview from left to right:

• The CLI chats about an image from the terminal, with 4-bit quantization keeping the 7B model under 8GB of VRAM.
• The Gradio web server renders the browser chat and polls the controller for newly launched models.
• The controller keeps a registry of GPU workers with heartbeats and dispatches by lottery or shortest queue.
• The model worker boots one checkpoint and streams generations, registering itself every fifteen seconds.
• The builder is load_pretrained_model, resolving merged, projector-only, unmerged-LoRA, and bitsandbytes-quantized layouts.
• The architecture mixin splits the token sequence at the image placeholder and interleaves image embeddings between the text gaps.
• The CLIP tower runs frozen at 336 pixels and hands over patch features from the second-to-last layer.
• The training script implements the two-stage recipe — 558K caption-alignment pairs, then the 665K-mixture visual instruction tuning set.

Why You Need This

First, this is the clearest existing lesson in what a multimodal LLM actually is. The answer, at the source level, is: a normal LLM whose input embedding matrix occasionally receives image features instead of token embeddings. In llava_arch.py, prepare_inputs_labels_for_multimodal finds every position where input_ids equals -200, embeds the surrounding text chunks as usual, and concatenates the projected image features into the gap — then rebuilds labels with IGNORE_INDEX over the image positions so the loss only trains on text. No new attention type, no fusion layer, no cross-attention. A 576-patch image becomes 576 soft tokens. Once you have read that function, the entire genre of vision-language models loses its mystique, because most of them are variations on this splice.

Second, the training recipe is a masterclass in doing less. Stage one connects the frozen tower to the frozen LLM by training only the projector on 558K caption pairs — the code enforces this with requires_grad flags, and the checkpoint that matters is a single mm_projector.bin. Stage two unfreezes the LLM and tunes on the 665K mixture with hyperparameters borrowed from Vicuna: global batch 256 at learning rate 1e-3 for pretrain, 128 at 2e-5 for finetune, one epoch each. The published wall-clock numbers make the point better than adjectives: pretraining takes 3.5 hours for the 7B on eight A100s, and instruction tuning about ten hours at 40GB or twenty at 80GB. The trainer adds two quietly clever touches — a modality-length-grouped sampler that batches image samples and text samples separately for a reported 25 percent speedup, and a separate learning rate for projector parameters. The paper’s efficiency was not hardware luck; it is visible as flags.

Third, the serving cluster is a miniature of production architecture. Controller, web server, and workers are separate processes talking over HTTP, with heartbeats every fifteen seconds and a thirty-second expiration that removes dead workers. Dispatch is a two-entry enum — lottery or shortest queue — and each worker reports its speed and queue length in a plain dataclass, so a new checkpoint becomes visible in the UI the moment it registers. The SGLang worker even shows how to route the same LLaVA conversation format to a high-throughput third-party backend. For anyone who has wondered how multi-model demo deployments keep a live model list, this is the smallest complete example we know of.

The detail view: the serving cluster's five processes, the bridge with its three LLM backends, the vision stack, the training machinery, and the evaluation harness.

The detail diagram’s bridge group shows the mixin pattern that keeps the model portable. LlavaMetaModel and LlavaMetaForCausalLM live once in llava_arch.py, and the three language backends — llava_llama, llava_mistral, llava_mpt — each mix them into their HuggingFace base class in about a hundred lines. The builder selects among them by substring match on the model name, and handles the awkward real-world layouts too: projector-only checkpoints that need a base model plus mm_projector.bin, unmerged LoRA adapters that are restored with non_lora_trainables.bin and merged on load, and 4-bit nf4 quantization through bitsandbytes with double quantization enabled.

The vision group rewards close reading too. clip_encoder.py wraps CLIPVisionModel in a class that never lets gradients through, selects the second-to-last hidden state, and drops the class token to keep patch features only. Its sibling CLIPVisionTowerS2 adds the scales-on-scales trick — running the tower at 336, 672, and 1008 pixels and merging — which improves the tower without touching the rest of the model. On the preprocessing side, mm_utils.py implements the anyres scheme: select_best_resolution scores candidate canvas sizes by effective versus wasted resolution, the image is divided into patches plus one global view, and unpad_image trims padding rows before the grid is flattened so wide and tall photos keep their true aspect ratio.

The training group is where the recipe becomes code. train.py holds five separate preprocess functions — one per conversation template — and LazySupervisedDataset, which defers tokenization until each sample is actually fetched and exposes modality_lengths for the grouped sampler. LLaVATrainer subclasses the HuggingFace Trainer to install that sampler, to give projector parameters their own learning rate, and to save projector-only checkpoints by gathering them out of ZeRO-3 shards. Around it sit a three-line train_mem.py entry, a FlashAttention monkey patch, and the zero3 and zero3_offload configs that the finetune script points at.

One serving file deserves its own paragraph because it quietly teaches prompt engineering internals. conversation.py is a catalog of chat templates as data: a Conversation dataclass with roles, messages, and separator styles for single-separator Vicuna, double-separator v1, LLaMA-2 instruction wrapping, MPT, and the plain format used in the first pretraining stage. The image placeholder is inserted into the first user message, and when the demo receives a picture it pads it to square with a fixed gray background color rather than cropping — the same pad choice that shows up in training as a mild hallucination reducer. Changing which template a model uses is a single conv_mode string, which is why the same weights can serve every backbone generation.

Evaluation is a fourth group worth exploring. run_llava.py is the one-call entry, model_vqa.py loops a question file against an image folder and writes answers, and the GPT-assisted review flow from the paper runs in three steps: generate answers, have GPT-4 score them against reference responses with a rule file, then summarize the verdicts into a table. Alongside sit per-benchmark evaluators — POPE, ScienceQA with its re-query variant, TextVQA — and a 301-line m4c_evaluator that parses the idiosyncratic answer formats of OCR and doc-style benchmarks. All twelve reported benchmarks are reproducible from scripts in this folder, with greedy decoding as the documented default.

From Install to First Chat

The README is blunt about platforms — Linux first, with separate docs for macOS and Windows — and the install is standard editable-package plus the training extra.

git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
python -m llava.serve.cli \
    --model-path liuhaotian/llava-v1.5-7b \
    --image-file "https://llava-vl.github.io/static/images/view.jpg" \
    --load-4bit

That last command downloads the 7B checkpoint from HuggingFace, quantizes it to 4 bits, and starts an interactive terminal session about the image you passed. Swap in gradio_web_server, controller, and model_worker invocations for the full browser demo, or use the load_pretrained_model function directly in Python for the one-call quickstart, which returns the tokenizer, model, image processor, and context length in a single tuple, ready for your own inference loop.

Honest limits: this is a research repository, not a product. The README’s install section warns off non-Linux systems in bold, full fine-tuning assumes eight A100s, and the main line’s later work — LLaVA-NeXT with its 4x-pixel anyres scaling, video support, and LLaMA-3 and Qwen backbones — moved to a separate codebase, leaving this one as the 1.5-era reference. The Apache-2.0 license covers the code, while checkpoints inherit the Llama community license and the instruction data carries OpenAI terms of use. Quantized inference trades some accuracy, and the eval suite’s twelve benchmarks use greedy decoding by design rather than beam search. None of that dulls the educational value: for the smallest complete proof that an LLM can learn to see, this remains the source to read.

Enjoyed this post? Never miss out on future posts by following us