Most inference stacks spend their lives downstream of a model. NVIDIA’s Model Optimizer — ModelOpt for short — lives upstream, where the model itself is made smaller. The repository, NVIDIA/TensorRT-Model-Optimizer on GitHub, is the source behind the nvidia-modelopt PyPI package: a library of compression techniques covering post-training quantization, quantization-aware training, weight and attention sparsity, structured pruning, distillation, and speculative decoding. Its README states the contract plainly: you bring a Hugging Face, PyTorch, or ONNX model; ModelOpt hands back an optimized, quantized checkpoint that downstream engines like vLLM, SGLang, TensorRT-LLM, and TensorRT can execute at reduced precision.

What makes the codebase worth a source tour is its architecture. Almost every technique is expressed as a mode — a self-describing object that knows how to rewrite a model, how to restore it, and how to serialize its own state. That uniformity is why a quantization call and a sparsification call look nearly identical in user code, even though one swaps nn.Linear for a fake-quantized replica and the other replaces dense attention with a masked kernel. It is also why the library can chain techniques — prune, distill, quantize, export — through one shared state manager.

This post walks the main branch as of October 2026. We map the package from its three public facades down through the quantization engine, the low-precision tensor implementations, the sparsity and pruning searchers, and the export layer that turns a quantized model into something an inference server can load. Every symbol we cite links to its actual line in the repository.

Overview: the three public namespaces (mtq, mts, mtp) sit on top of a shared mode framework; the quantization core rewrites layers and fills calibrator statistics; results flow into the Hugging Face export and the model-free ONNX path.

Reading the overview from left to right:

• mtq namespace, L26 — the quantization facade imports compress, config, and conversion submodules into one flat namespace, so mtq.quantize(...) is a one-liner.
• mtq.quantize, L310 — the central PTQ entry point that takes a model, a config dict, and a forward_loop.
• INT8_DEFAULT_CFG, L1699 — preset configurations are loaded from YAML files shipped in the modelopt_recipes package and exposed as module constants.
• QuantModuleRegistry, L203 — the registry that maps ordinary nn.Module types to their quantized replicas.
• _Calibrator, L25 — the base class behind max, entropy, and MSE calibration strategies.
• mts.sparsify, L32 — the weight-sparsity entry point driving one-shot mask searchers.
• mtp.prune, L31 — the structured-pruning entry that searches over width and depth choices.
• export_hf_checkpoint, L1871 — the unified Hugging Face export that writes quantized checkpoints deployable in vLLM and SGLang.
• quantize, L122 — the model-free ONNX PTQ path, for when you have a graph rather than a training-capable model.

One Facade, Five Modes

The spine of the library is modelopt/torch/opt/mode.py. ModeDescriptor, L56 declares what a technique is called, which configuration schema it accepts, and how it should be applied; a private registry class at L277 keeps every descriptor keyed by name. Each technique module defines its own descriptor — QuantizeModeDescriptor, L84 for quantization, one for sparsity, one for distillation — and registers it with the shared machinery at import time.

The payoff is a uniform lifecycle in modelopt/torch/opt/conversion.py. ModeloptStateManager, L64 tracks which modes are currently active on a model; apply_mode, L343 is the single funnel every facade call goes through; and save, L495 serializes mode state alongside weights so a quantized checkpoint remembers how it was made. When quantization/conversion.py’s convert_to_quantized_model, L63 folds fake quantizers into real packed weights, it does so as one conversion step in this same lifecycle, not as a side door.

Five facades share this spine: mtq (quantization), mts (sparsity), mtp (pruning), modelopt.torch.distill (knowledge distillation, whose convert, L32 attaches a teacher for logit matching), and modelopt.torch.speculative (convert, L30, which bolts draft models onto a target for multi-token decoding). Distillation and speculative decoding are also how the library recovers accuracy after aggressive compression — prune a model, then teach the pruned student to behave like the original.

PTQ: Calibrate, Quantize, Convert

Post-training quantization in ModelOpt is a three-act flow. Act one: mtq.calibrate — calibrate, L73 — runs your data through the model while calibrators watch activation ranges. The calibration hierarchy starts at _Calibrator, L25; MaxCalibrator, L26 tracks a running maximum, while HistogramCalibrator, L35 builds histograms that entropy- and MSE-based clipping strategies search for better saturation points.

Act two is the rewrite. mtq.quantize walks the module tree and consults QuantModuleRegistry, L203, swapping each recognized layer class for a quantized replica that owns two children: a weight quantizer and an input quantizer, both instances of TensorQuantizer, L136. During the quantized forward pass each quantizer delegates to the autograd functions in tensor_quant.py — FakeTensorQuantFunction, L332 for integer formats, ScaledE4M3Function, L417 for FP8 — which simulate rounding without changing dtype. That simulation is what makes quantization-aware training possible: gradients flow through the fake-quant math, so a few finetuning steps can restore accuracy that pure PTQ would lose.

Act three is conversion and selection. For deployment you call the real-quantization conversion to pack weights and scales into deployable form; for hard cases there is auto_quantize, L590, which searches over per-layer format assignments under a memory or throughput budget — the mode behind it, AutoQuantizeModeDescriptor, L208, scores candidates by sensitivity, not exhaustive evaluation. A KV-cache specialist, kv_cache_auto_quant.py, searches per-layer KV formats automatically.

From Install to Your First Model

Installation is one line from the README:

pip install -U nvidia-modelopt[all]

The [all] extra pulls in the framework plugins (Hugging Face Transformers, diffusers, and friends) that ModelOpt discovers lazily at import. The canonical quantization snippet below is taken verbatim from the module docstring of modelopt/torch/quantization/config.py (lines 132–135), which is the same pattern every example in the repository uses:

import modelopt.torch.quantization as mtq

def forward_loop(model):
    # feed representative calibration data through the model
    for batch in calib_dataloader:
        model(**batch)

model = mtq.quantize(model, mtq.INT8_DEFAULT_CFG, forward_loop)

Three things to notice. First, quantization never touches your training code — the only new artifact is forward_loop, a plain callable that runs calibration batches. Second, the config is data: mtq.INT8_DEFAULT_CFG is a dictionary loaded from modelopt_recipes/configs/ptq/presets/model/int8.yaml, so you can deepcopy and edit it — the same docstring shows the custom-config pattern, starting from mtq.INT4_AWQ_CFG and appending an entry that disables the *lm_head* quantizer. Third, because quantization is a registered mode, mtq.quantize(model, cfg, loop) is equivalent to applying the mode through the shared state manager, so modelopt_state/save round-trips work for free. For a first experiment, a small model is enough: run the snippet on any transformers causal LM, inspect a layer (it now carries weight_quantizer and input_quantizer attributes), and call the real-quantization conversion to see packed weights.

FP8 and NVFP4: Low-Precision Paths

INT8 is the baseline, but the interesting files in this repository are the floating-point formats. Presets like FP8_DEFAULT_CFG, L1706 and NVFP4_DEFAULT_CFG, L1740 configure the same quantizer machinery for E4M3 and 4-bit floating point with two-level block scales. On the tensor side, real quantization produces packed objects deriving from BaseQuantizedTensor, L40: FP8QTensor, L33 for per-tensor FP8, and NVFP4QTensor, L67 for NVFP4, where weights are grouped into blocks of sixteen with a per-block E4M3 scale and a per-tensor FP8 scale above it. Sibling files cover MXFP8, MXFP4, INT4, and NF4 — one module per format, one dequantize contract each.

The preset YAMLs under modelopt_recipes/configs/ptq/presets/model/ are worth reading even if you never quantize: nvfp4.yaml shows exactly which module classes get weight-only versus weight-and-activation treatment, and the recipe directory also carries IQ1/IQ2-style entries for extreme weight-only compression. QAT enters through the same facade — quantized replicas are already gradient-carrying, so mts-style mode application plus an optimizer loop is all a training harness needs; NVIDIA’s QAT blog series from 2025 describes this workflow in detail for gpt-oss-class models.

One export detail deserves mention for serving: attention itself needs special handling at FP8, which is why export_onnx.py contains export_fp8_mha, L356 alongside the generic export_int8, L132 and export_fp8, L253 — fused multi-head attention has to survive graph surgery as a unit rather than being decomposed into Q/DQ pairs.

Sparsity, Pruning, and Export

The compression family beyond quantization follows the same shape. Weight sparsity exposes mts.sparsify, which drives searchers such as SparseGPTSearcher, L144 — a one-shot, Hessian-based mask search that zeroes weights and reconstructs the remaining ones to minimize output error — and MagnitudeSearcher for the simpler magnitude criterion. Attention sparsity is a different beast entirely: SparseAttentionModule, L27 replaces dense attention with token-selection logic, and the plugin in plugins/vllm.py implements ModelOptSparseAttentionBackend, L801, a drop-in flash-attention backend so a sparsified model can actually run on the vLLM server rather than merely export to it.

Pruning is search again: mtp.prune, L31 evaluates width and depth candidates, with FastNASModeDescriptor, L327 providing the channel-search mode and a binary searcher that halves or keeps blocks. The practical recipe, demonstrated in NVIDIA’s Minitron work, is prune → distill → quantize — one mode call per stage.

The export layer is where checkpoints meet serving engines. export_hf_checkpoint, L1871 writes a Hugging Face-compatible quantized checkpoint that vLLM, SGLang, and TensorRT-LLM load natively; capture_quantized_weight_export_state, L375 snapshots the packed weights and scales during that process, and ExportContext, L44 dispatches per-model handlers. For deployment to classic TensorRT engines there is the ONNX path: the model-free quantizer at quantize, L122 inserts Q/DQ into an existing graph, its FP8 sibling at quantize, L140 can upgrade an INT8 graph in place via int8_to_fp8, L56, and the graph-surgery CLI at main, L64 handles the finicky edits (DQ-transpose folding, GQA replacement) that raw ONNX tooling makes painful. If your serving stack is ONNX-centric, our ONNX overview pairs well with this section.

Detail: the quantization core (config, mode, registry, fake-quant kernels) feeds both the real-quantization qtensor formats and the ONNX writers; sparsity searchers and the pruning mode converge on the same unified export that emits quantized checkpoints and TRT-LLM configs.

Honest Limits

The source is honest about what it cannot do, and so should we be. First, layout drift is real: older write-ups (including some NVIDIA docs) reference a modelopt/quantization/ package of standalone low-precision kernels, plus modelopt/core/ and modelopt/experimental/ directories — none of those exist on the current main; the low-precision tensor implementations now live under modelopt/torch/quantization/qtensor/, and preset configs moved into the separate modelopt_recipes package. Second, naming drift: mtq.int8-style preset attributes from 2024 tutorials are gone; the constants are INT8_DEFAULT_CFG and friends. Third, hardware: FP8 kernels target Hopper and newer, NVFP4 targets Blackwell; on unsupported GPUs the fake-quant simulation runs, but you are measuring accuracy potential, not speed. Fourth, accuracy is never guaranteed — INT8 SmoothQuant is usually free, W4A16 AWQ is usually close, but W4A4 NVFP4 and 2:4 sparsity are trade-offs where QAT or quantization-aware distillation is effectively mandatory. Finally, the library is pre-1.0 with a one-release deprecation window, so pin your version. Few compression toolkits make their entire surface — calibrators, fake-quant autograd, sparse attention kernels, graph surgery — this readable.

Enjoyed this post? Never miss out on future posts by following us