Segment Anything Model 2, released by Meta’s FAIR lab as the facebookresearch/sam2 repository, is the successor to the original Segment Anything model, and its most important idea is unification: one promptable segmentation model handles both still images and video. You give it clicks, bounding boxes, or a rough mask; it returns high-quality object masks. On images that is a single forward pass. On video, the model treats every frame as a new observation of the same object and tracks the mask forward and backward through the clip, so annotating one object once propagates through the whole sequence. That single design decision is why SAM 2 became the default tool for fast visual annotation and why its demo drew so much attention when it landed.

What makes the source worth reading is the streaming memory architecture under the hood. Most video segmentation systems of its generation leaned on optical flow between frames or on full-graph optimization after the fact. SAM 2 does neither: as it processes frames one at a time, it stores compact spatial memories of what it has already seen — the predicted mask, downsampled and fused with backbone features — and conditions each new frame on that memory bank through cross-attention. There is no second pass over past frames, no expensive temporal smoothing stage, and still the masks stay temporally consistent. Understanding how that loop is wired means reading sam2/modeling/sam2_base.py and the modules it coordinates, which is exactly what this tour does.

To make the structure concrete, we generated two architecture diagrams directly from the repository tree — a compact overview and a detail view that goes down to individual methods. Below we walk the overview left to right, then zoom in, then run the model from a fresh install, and finish with an honest look at the failure modes you will hit in production use.

SAM 2 at a glance: three entry points funnel into SAM2Base, which coordinates the Hiera vision stack, the promptable SAM heads, and the memory modules that carry masks across frames.

Reading the overview from left to right:

• Everything starts at build_sam2, L71, a Hydra-driven factory that composes a model config, instantiates the module tree, and loads the checkpoint. Its sibling build_sam2_video_predictor, L100 builds the same weights but swaps in the video predictor class.
• On the user side, SAM2ImagePredictor, L20 is the image facade: set_image, L86 embeds the picture once so subsequent prompts are cheap, and predict, L237 turns clicks or boxes into masks. For prompt-free operation, SAM2AutomaticMaskGenerator, L36 grids the image with point prompts and post-processes the result.
• For video, SAM2VideoPredictor, L19 manages an inference_state dict: init_state, L42 loads and resizes the frames, add_new_points_or_box, L161 registers a prompt on any frame, and propagate_in_video, L546 streams masks through the whole clip.
• Both predictors delegate to SAM2Base, L22, the real engine. Its per-frame method track_step, L814 is the beating heart of the propagation chain — everything else in the diagram exists to serve it.
• The vision side is ImageEncoder, L14, which wraps the Hiera, L169 hierarchical ViT trunk and an FPN neck to emit multi-scale feature maps.
• The promptable heads are PromptEncoder, L17, which embeds clicks, boxes, and masks into sparse and dense tokens, and MaskDecoder, L15, which runs TwoWayTransformer, L19 bidirectional attention before upsampling to mask logits.
• The memory side is what distinguishes SAM 2 from its predecessor: MemoryEncoder, L138 writes new memories after each frame, and MemoryAttention, L102 conditions current features on everything remembered so far.

What the Detail Diagram Adds

The overview hides the mechanics inside SAM2Base’s one-frame cycle, so the second diagram expands the same repository into thirty-plus nodes grouped by responsibility — assembly, streaming core, SAM heads, vision backbone, memory stack, and support utilities. It is worth a slow scroll before reading the next sections.

The detail view adds per-method nodes: track_step orchestrates memory selection, head decoding, and memory writing for every frame of the propagation chain.

Three clusters deserve attention. First, the streaming core: track_step calls _prepare_memory_conditioned_features to gather memory, _forward_sam_heads to decode masks, and _encode_new_memory to write new memory — a read-decode-write loop per frame. Second, the heads: the mask decoder’s predict_masks path runs the two-way transformer whose cross-attention blocks use RoPEAttention with rotary position embeddings, a detail borrowed from modern LLMs. Third, support: load_video_frames spins up an AsyncVideoFrameLoader that prefetches frames on a background thread so decoding latency hides behind GPU compute, and fill_holes_in_mask_scores cleans disconnected components out of predicted masks.

From Install to Your First Model

SAM 2 targets Python 3.10+ and PyTorch 2.3.1 or newer. From a clone of the repository, an editable install pulls in the handful of dependencies (no CUDA extension compilation is required for inference; building one is optional for training speed):

git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e .

Checkpoints live in the repo’s release and on the Hugging Face hub, where the model identifiers map to config-plus-weight pairs — facebook/sam2.1-hiera-large being the highest-quality variant, with tiny, small, and base-plus siblings trading accuracy for speed. The simplest image workflow skips explicit config handling entirely via from_pretrained:

from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2.1-hiera-large")
predictor.set_image(image)  # np.ndarray (H, W, 3) RGB or PIL image
masks, scores, logits = predictor.predict(
    point_coords=[[600, 400]], point_labels=[1]
)

predict is cheap after set_image because the expensive embedding is computed once and cached; you can add negative clicks with label 0 iteratively until the mask is right. The video workflow mirrors the diagram’s state machine:

from sam2.build_sam import build_sam2_video_predictor

predictor = build_sam2_video_predictor(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "checkpoints/sam2.1_hiera_large.pt",
)
state = predictor.init_state("videos/your_clip.mp4")
predictor.add_new_points_or_box(
    state, obj_id=1, points=[[600, 400]], labels=[1]
)
for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
    pass  # masks per object per frame, streamed as the loop advances

Two practical notes. You can jump to any frame and add new prompts mid-propagation; the model re-runs affected frames rather than everything. And each object gets its own obj_id, so multi-object tracking is a first-class API — though cost grows with object count, as we will see below.

Inside the Streaming Memory Loop

The design question at the core of SAM 2 is: what should a frame know about the past? The answer is a bounded memory bank per object. When SAM2Base, L22 processes frame t, its _prepare_memory_conditioned_features path first asks select_closest_cond_frames (from sam2/modeling/sam2_utils.py) which past frames are worth attending to: every frame where the user prompted (conditioning frames), plus a sliding window of recent frames. Those stored memories — spatial feature maps fused with downsampled predicted masks, plus a single-vector object pointer per memory — are concatenated and fed to the memory cross-attention.

The decoder side then runs as in image segmentation, but with one twist: because memory can make the single best mask ambiguous, the model applies a stability-based fallback, re-picking among multiple mask hypotheses when IoU confidence is low. This dynamic_multimask_via_stability behavior is toggled on by the builder’s default override arguments you saw referenced near build_sam2. After the mask is chosen, _encode_new_memory pushes the frame’s features and the predicted mask through the memory encoder, and the loop advances by one frame. Nothing about the past is recomputed — the propagation chain is strictly O(frames) once prompts are placed.

This is also why the architecture is genuinely streaming: future frames never need to exist when processing frame t, which is what enables the interactive loop of the demo, where you scrub backward, re-prompt, and let only the affected span re-propagate.

Where SAM 2 Fits in the Ecosystem

If you have followed our earlier tours, the lineage is easy to place. The original Segment Anything established the promptable-mask paradigm and the SA-1B dataset, but treated video by per-frame segmentation with no persistence. SAM 2 keeps the data-engine mindset and adds the memory loop, achieving better accuracy with roughly one-sixth the interactive annotations in their ablations. The Hiera backbone itself descends from the MAE-style pretraining line, and if you want to experiment with alternative trunks, the image-encoder wrapper is a thin, clean seam to modify. For classical detection and instance segmentation baselines, Detectron2 remains the reference toolkit, and SAM 2 outputs can be dropped next to its box and mask heads for comparison studies. All of this stands on PyTorch, and the codebase is a pleasant specimen of a research repo: plain nn.Modules, a single Hydra config tree per model variant, and no metaclass magic.

Honest Limits

• Confusable objects. Tracking relies on appearance plus motion consistency, not identity reasoning. Two visually similar objects crossing paths can swap masks mid-video; the classic failure demo is occlusion followed by re-emergence in a slightly different pose.
• Memory costs scale with objects and video length. Each object holds its own memory bank on GPU, and long clips accumulate per-frame outputs in inference_state. For long videos with many objects, memory pressure is real; the code offers a vos_optimized build with torch.compile, but the fundamental scaling remains.
• No semantics. Masks are class-agnostic. SAM 2 will happily segment a shadow or a reflection if prompted; attaching labels, relationships, or filtering by category is your job.
• Prompt sensitivity in degenerate scenes. Extremely small objects, heavy motion blur, or duplicated textures can yield unstable masks in the first frames, and errors early in the propagation chain persist until you intervene with a corrective prompt.
• Fixed input resolution. Images are resized to 1024 on the long side by the internal transforms, and video frames to a configured shorter-side size, so sub-pixel boundary fidelity on tiny structures has a ceiling.
• Checkpoint and license logistics. Weights are released under their own model license (Apache 2.0 for code) and need to be downloaded separately or pulled from the hub — worth automating for reproducible environments.

None of these are disqualifying — they are the edges of a design that chose streaming speed and interactivity over exhaustive global optimization. Reading the source makes those trade-offs explicit, and once you can trace a click through prompt encoding, two-way attention, memory write, and cross-frame readback, SAM 2 stops being a black box and becomes a component you can confidently wire into larger vision systems.

Enjoyed this post? Never miss out on future posts by following us