Detectron2 is Meta AI Research’s next-generation library for object detection and segmentation, and it is the direct successor of the original Detectron and of maskrcnn-benchmark. Written in PyTorch, it has carried a whole generation of computer vision research: Faster and Mask R-CNN, panoptic segmentation, DensePose, Cascade R-CNN, PointRend, ViTDet, and a long tail of projects built on top of it. Unlike many model dumps that stop at weights and a training script, Detectron2 ships a genuinely readable reference implementation of the entire two-stage detection stack, which is why it remains one of the best codebases to study if you want to understand how modern detectors really work.

What makes the source so approachable is how small the load-bearing core is. One meta-architecture class, GeneralizedRCNN, L25, owns the end-to-end flow: it holds a backbone, a proposal generator, and ROI heads, and it simply wires them together. Everything else is a registry-driven factory: you pick a backbone by name, a proposal generator by name, ROI heads by name, and a yacs CfgNode tree supplies every constant, threshold, and channel count. That registry pattern is also why so many papers ship a Detectron2 config: the architecture description is data, not code. This tour reads that machinery as it exists on the main branch today, file by file, function by function.

Two diagrams accompany this post. The overview shows the ten-ish components you actually need to hold in your head; the detail graph zooms into the individual functions and their call relationships, from config ingestion down to anchor tiling and box decoding. Both link to real files in the repository.

Overview of Detectron2: the GeneralizedRCNN meta-architecture assembles a ResNet (optionally wrapped by FPN) backbone, an RPN proposal stage built on DefaultAnchorGenerator, and StandardROIHeads; DefaultPredictor and DefaultTrainer drive it, while the yacs config, data loaders, and DetectionCheckpointer form the surrounding plumbing.

Reading the overview from left to right:

• GeneralizedRCNN, L25 — the single nn.Module that owns the whole detection stack and branches between training forward and eval inference.
• build_backbone, L20 — the factory that turns cfg.MODEL.BACKBONE.NAME into a concrete backbone, optionally wrapped with FPN.
• ResNet, L362 — the classic stem-plus-four-stages backbone producing C2–C5 feature maps.
• FPN, L17 — the Feature Pyramid Network that merges those stages into multi-scale outputs.
• DefaultAnchorGenerator, L86 — tiles anchor boxes of configured sizes, ratios, and angles at every location of every feature level.
• RPN, L181 — scores and regresses those anchors into a short list of object proposals.
• StandardROIHeads, L530 — pools features on each proposal and emits classes, refined boxes, and optional masks.
• DefaultPredictor, L284 — the one-image, one-device convenience wrapper most people meet first.

From Install to Your First Model

Installing Detectron2 means compiling a few CUDA/C++ ops against your exact PyTorch build, so the version match matters more than with a pure-Python library. The canonical path from the project’s docs is:

pip install 'git+https://github.com/facebookresearch/detectron2.git'

Assuming a matching torch and CUDA toolchain are already present, that single command builds the extensions and drops the detectron2 package into your environment. The fastest way to a first result is the DefaultPredictor, L284:

import cv2
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor

cfg = get_cfg()
cfg.merge_from_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"
cfg.MODEL.DEVICE = "cpu"  # skip this line if you have a GPU

predictor = DefaultPredictor(cfg)
im = cv2.imread("input.jpg")
outputs = predictor(im)["instances"]
print(outputs.pred_boxes, outputs.pred_classes, outputs.scores)

Three lines inside DefaultPredictor.__init__ tell you the whole runtime story: build_model(self.cfg) constructs the network, DetectionCheckpointer(self.model).load(cfg.MODEL.WEIGHTS) pulls the weights (the detectron2:// scheme resolves to the model zoo CDN), and a ResizeShortestEdge transform fixes the input geometry. __call__ then converts BGR to the configured input format, resizes, and calls the model on a one-element batch. On the training side, DefaultTrainer, L355 is the opinionated counterpart: it builds the optimizer, wraps the data loaders, registers checkpoint and evaluation hooks, and inherits the hook machinery of TrainerBase, L94. For multi-GPU runs, launch, L27 either spawns one process per GPU or submits a Slurm job before invoking your main.

GeneralizedRCNN: The Meta-Architecture Glue

Everything starts at build_model, L16, which resolves cfg.MODEL.META_ARCHITECTURE against a registry and calls the class’s from_config. For GeneralizedRCNN, from_config, L72 does exactly three builds — backbone, proposal generator, and ROI heads — and returns them as constructor kwargs. There is no hidden state and no base-class magic; the model is literally the composition of those three submodules plus two registered buffers, pixel_mean and pixel_std.

The training path lives in forward, L126: normalize images via preprocess_image, L223, run self.backbone(images.tensor) to get feature maps, ask the proposal generator for proposals plus its losses, then hand features, proposals, and ground truth to the ROI heads for the detector losses. The method merges both loss dicts and returns them — no loss computation logic of its own. At eval time forward simply delegates to inference, L178, which runs the same chain without ground truth and finishes with _postprocess, L237, rescaling predicted boxes from the resized input back to the original image resolution so users never see coordinates tied to network input geometry.

Backbone and FPN: Multi-Scale Features

build_backbone, L20 reads cfg.MODEL.BACKBONE.NAME — build_resnet_backbone by default — and, when cfg.MODEL.FPN.IN_FEATURES is non-empty, wraps the result so the model consumes pyramid features instead of a single scale. The builder build_resnet_backbone, L614 maps RESNETS.DEPTH (50, 101, 152) to stem and stage specifications and assembles the ResNet, L362 object, whose make_stage, L606 helper stacks bottleneck blocks while threading input channels into output channels.

The FPN, L17 module adds 1x1 lateral convolutions over the chosen C2–C5 inputs, applies top-down upsampling, and merges the two paths with a FUSE_TYPE operation (sum or average). Its output is a dict of p2 through p6 maps, and output_shape() reports each level’s stride and channels — metadata that the anchor generator and ROI pooler later query. This division of labor is typical of the codebase: the backbone produces features, but everything downstream stays agnostic by consuming the output_shape() contract.

Proposals: Anchors, RPN Head, Top-k Selection

The proposal side is where Detectron2’s config-driven style shines. DefaultAnchorGenerator, L86 precomputes cell anchors from ANCHOR_GENERATOR.SIZES and ASPECT_RATIOS and shifts them across every position of every feature level, returning per-level anchor boxes as buffers. The RPN, L181 module then applies the shared StandardRPNHead, L67 — a 3x3 conv stack with 1x1 objectness and box-delta outputs — to each level, and passes objectness-sorted candidates through find_top_rpn_proposals, L22, which decodes deltas, clips to image bounds, drops tiny boxes, applies NMS, and keeps PRE_NMS_TOPK/POST_NMS_TOPK proposals per image (12000/2000 in training, 6000/1000 at test with defaults). Construction itself goes through build_proposal_generator, L15, so swapping RPN for a learned-proposal variant is a one-line config change.

ROI Heads and the Training Loop

Proposals become detections in the ROI stage. build_roi_heads, L38 resolves cfg.MODEL.ROI_HEADS.NAME, and StandardROIHeads, L530 implements the canonical flow: an ROIAlignV2 pooler crops each feature level at proposal coordinates, a box head classifies and refines, and optional mask and keypoint branches predict per-ROI pixel outputs. The classification edge is FastRCNNOutputLayers, L174, which owns class scores, box regression, the smooth-L1 loss, and score-threshold plus NMS inference; the mask branch is MaskRCNNConvUpsampleHead, L215, a small stack of 3x3 convs with 2x upsampling ending in a 28x28 per-class mask.

During training, StandardROIHeads samples a balanced subset of proposals against ground truth (BATCH_SIZE_PER_IMAGE, POSITIVE_FRACTION), computes box and mask losses, and returns them for forward to merge. The plumbing around this loop is equally explicit: build_detection_train_loader, L521 builds an infinite, aspect-ratio-grouped loader whose DatasetMapper applies augmentation per image, and build_detection_test_loader, L621 gives a deterministic eval loader. Persistence is centralized in DetectionCheckpointer, L16, which extends the generic Checkpointer with model-compatibility fixes and conversion shims for older Detectron and Caffe2 weights — the reason cfg.MODEL.WEIGHTS can point at files produced years ago by different frameworks.

The detail view traces the call graph: from_config builds the three children; ResNet stages feed FPN levels; anchors and the RPN head feed find_top_rpn_proposals; ROI heads pool, classify, and mask; and DefaultPredictor/DefaultTrainer wrap everything with checkpointing and data loaders.

Honest Limits

Detectron2 is explicitly research-oriented, and the repository is in maintenance mode: it is stable and widely used, but expect feature work to happen in successor projects rather than here. Treat it as a reference implementation and an experiment bench, not as a batteries-included serving framework — there is no first-class REST serving story, and production deployments typically export to TorchScript or ONNX or rebuild inference elsewhere.

The build is CUDA-centric. Custom ops compile against a specific torch/CUDA pairing, so mismatches produce long, cryptic build failures, and CPU-only or Windows setups remain awkward. Pin your torch version first, then install Detectron2 against it. None of this blocks study or experimentation — it just means you should budget setup time before the interesting part starts.

Config handling has two coexisting eras. The classic yacs yaml system — the CfgNode defaults merged with merge_from_file — powers all legacy model zoo configs, while the newer LazyConfig (plain Python objects) is the recommended path for new work. Docs and tutorials mix the two, so check which system a given config file belongs to before editing it. Finally, several model zoo and documentation links have been retired or redirected over the years; checkpoint URLs from old tutorials may need manual repair, and the detectron2:// scheme depends on CDN paths that can move. Verify weights links before wiring them into automation.

If you want the contrast, single-stage detectors tell a different story — see our yolov5 tour for a one-stage design — while the tensor-level machinery underneath Detectron2 overlaps heavily with torchvision ops, runs on the pytorch runtime, and can be exported toward onnx for deployment.

Enjoyed this post? Never miss out on future posts by following us