Most deep-learning frameworks you meet day to day are executors: they take a graph, schedule its ops, and hope the kernels underneath are fast. TensorRT is a different animal. It is a compiler: it ingests a trained network, picks concrete CUDA kernels for every layer by timing candidates on your actual GPU, fuses what it can, and emits a self-contained, version-locked binary called an engine. The OSS repository we are touring today — NVIDIA/TensorRT, currently at version 11.4 — contains the public C++ API headers, the reference plugin library, the ONNX ingest layer, and the build scaffolding; the closed-source libnvinfer binary does the heavy lifting underneath them. Reading these headers is the best way to understand what the compiler promises you, which is exactly what this post walks through.

If you have followed our earlier tours of ONNX as the interchange format and of TensorRT-LLM as the LLM-focused sibling project, this is the foundation those posts stood on. TensorRT consumes ONNX as its primary input — the parser that converts a protobuf graph into TensorRT’s own INetworkDefinition lives right in this repository (well, its header does; the implementation is the onnx-tensorrt submodule). Everything else — tactic selection, plugin resolution, serialization — flows from the two dozen interfaces declared in include/.

The codebase is organized around a clean split. The build side (include/NvInfer.h) owns network construction and engine compilation; the runtime side (include/NvInferRuntime.h) owns deserialization and execution; the plugin side (include/NvInferPluginBase.h, include/NvInferRuntimeCommon.h, and the entire plugin/ tree) lets you extend the compiler with custom layers. Every public handle forwards to an opaque mImpl pointer, so the ABI can evolve without breaking recompiles — you will see the same VImpl pattern in ONNX Runtime and other ABI-sensitive projects.

TensorRT end to end: the ONNX parser fills an INetworkDefinition, the builder compiles it into a serialized engine plan, the runtime deserializes it into an ICudaEngine, and the plugin ecosystem extends the compiler with custom ops registered into a global registry.

Reading the overview from left to right:

• IParser, L191 — the ONNX front-end interface: parse(), parseFromFile(), and error reporting are the whole contract.
• createParser, L755 — the factory that binds a parser to a network definition and a logger; from here on the parser writes layers directly into your network.
• INetworkDefinition, L8168 — the framework-neutral graph TensorRT actually optimizes: tensors, layers, and marked outputs.
• buildSerializedNetwork, L11739 — the compiler entry point on IBuilder: it times tactics, builds profiles, and returns a serialized engine plan as IHostMemory.
• IRuntime, L1900 — the runtime-side factory; deserializeCudaEngine turns the plan back into a live engine.
• deserializeCudaEngine, L2032 — where the serialized blob becomes an executable object again, plugins resolved from the registry.
• ICudaEngine, L3143 — the immutable, per-GPU compiled artifact; it can spawn any number of execution contexts.
• IExecutionContext, L4089 — the mutable per-stream state: set input shapes, bind device addresses, call enqueueV3.
• IPluginRegistry, L55 — the global creator table both the builder and the runtime consult when a layer name has no built-in implementation.

The Build Side: Network, Builder, Config

Everything starts with two factory calls. createInferBuilder (NvInfer.h, L11916) gives you an IBuilder, and createNetworkV2 (NvInfer.h, L11662) mints an empty INetworkDefinition. In TensorRT 11 the explicit-batch flag that dominated 10.x documentation is gone — dynamic shapes are simply how the network works, described through optimization profiles rather than a creation flag.

Programmatically constructing a network is pleasingly mechanical: every layer type — convolutions, matmuls, elementwise ops, shuffles — is one method on INetworkDefinition, each returning a typed layer handle whose inputs and outputs are ITensor objects. You wire tensors together, then call markOutput (NvInfer.h, L8222) on whatever tensors escape the network. In practice almost nobody hand-builds graphs anymore; the ONNX parser does it, and (as the diagram shows) the parser is just another producer of the same graph IR.

The interesting object is IBuilderConfig (NvInfer.h, L10622). This is where precision flags (setFlag at NvInfer.h, L10725 for FP16, INT8, FP8, sparsity), workspace budgets via setMemoryPoolLimit (NvInfer.h, L11081), profiles, and DLA targeting all live. One builder can serve many configs; one config fully determines how aggressive the compiler gets. Memory is a first-class concern — the pool limits bound tactic workspace, persistent cache, and weight-streaming pools separately, which matters enormously once you target embedded GPUs where every megabyte is contested.

The detail view: how the ONNX ingest chain, the build/tactics surface, the runtime execution trio, and the plugin ecosystem connect — one node per source file, every node linked to the real code.

Tactics and the Timing Cache

“Tactic” is TensorRT’s word for a concrete kernel-scheduling choice for one layer: which convolution algorithm, which fusion grouping, which memory layout. At build time the compiler enumerates plausible tactics per layer and times each one on your GPU, keeping the winner. The knobs you control are TacticSource bits (NvInferRuntime.h, L2885) — in 11.x just kEDGE_MASK_CONVOLUTIONS and kJIT_CONVOLUTIONS remain, toggled through setTacticSources (NvInfer.h, L10981). The header comment on the JIT source is refreshingly honest: enabling it “may increase” engine building time.

Because re-timing hundreds of tactics on every build is painful, TensorRT persists the measurements. createTimingCache (NvInfer.h, L11016) returns an ITimingCache (NvInfer.h, L10209) that you attach to the config; after a build, ITimingCache::serialize (NvInfer.h, L10223) hands you a blob you can check in next to your model. CI farms love this: build once per driver/GPU family, replay the cache everywhere. The catch is documented right in the header — a cache “must be generated by a build of exact same version,” so caches, like engines, are version-locked artifacts rather than durable data.

The end product of all this measurement is not a live object but bytes: buildSerializedNetwork (NvInfer.h, L11739) returns IHostMemory containing the plan. Engines are only consumable by the same TensorRT version on the same compute capability — a deliberate trade: the compiler freezes GPU-specific decisions in exchange for not shipping a kernel-selection runtime. Serialization is also how the ecosystem moves engines around; serving stacks like Triton Inference Server treat the plan file as the deployable unit, exactly the role an .onnx file plays for ONNX Runtime.

From Install to Your First Model

The README’s fast path is one command: pip install tensorrt — the Python package wraps the same C++ API, so the concepts below transfer one-to-one. For the full OSS build you need CUDA 12.9 or 13.4, CMake ≥ 3.31, then git clone -b main https://github.com/nvidia/TensorRT TensorRT && git submodule update --init --recursive, and the release tarballs from NVIDIA provide libnvinfer plus the trtexec CLI you can use to sanity-check engines without writing code.

Here is the whole build-to-execute flow in C++, using only interfaces we verified in the tree above:

#include "NvInfer.h"
#include "NvOnnxParser.h"

using namespace nvinfer1;

Logger logger;  // your ILogger implementation

// Build side: compile model.onnx into a serialized engine plan.
auto builder = std::unique_ptr<IBuilder>(createInferBuilder(logger));
auto network = std::unique_ptr<INetworkDefinition>(builder->createNetworkV2(0));
auto parser  = std::unique_ptr<IParser>(nvonnxparser::createParser(*network, logger));
parser->parseFromFile("model.onnx", static_cast<int>(ILogger::Severity::kWARNING));

auto config = std::unique_ptr<IBuilderConfig>(builder->createBuilderConfig());
auto plan   = std::unique_ptr<IHostMemory>(
    builder->buildSerializedNetwork(*network, *config));

// Runtime side: deserialize and execute on a CUDA stream.
auto runtime = std::unique_ptr<IRuntime>(createInferRuntime(logger));
auto engine  = std::unique_ptr<ICudaEngine>(
    runtime->deserializeCudaEngine(plan->data(), plan->size()));
auto context = std::unique_ptr<IExecutionContext>(engine->createExecutionContext());
// set tensor addresses, then: context->enqueueV3(stream);

Every symbol in that snippet is real: createInferBuilder (NvInfer.h, L11916), createNetworkV2 (NvInfer.h, L11662), createBuilderConfig (NvInfer.h, L11635), parseFromFile (NvOnnxParser.h, L222), and on the runtime side createInferRuntime (NvInferRuntime.h, L5226), deserializeCudaEngine (NvInferRuntime.h, L2032), and createExecutionContext (NvInferRuntime.h, L3301). Actual GPU work is submitted asynchronously by enqueueV3 (NvInferRuntime.h, L4791) on a stream you own — there is no blocking execute dance anymore, and multi-stream serving is just more contexts on more streams.

The Plugin Registry and IPluginV3

When the parser meets an ONNX op TensorRT does not implement, or when you have a hand-written CUDA kernel no graph rewrite can express, plugins are the escape hatch. The current contract is IPluginV3 (NvInferPluginBase.h, L205), a versioned interface whose key move is the capability split: a plugin object returns separate interfaces for the kCORE, kBUILD, and kRUNTIME phases. Build-phase code answers “what output shapes and formats can you support”; runtime-phase code does the actual launch. The cleanest example in the tree is ROIAlignV3 (roiAlignPlugin.h, L58), which inherits IPluginV3, IPluginV3OneCore, IPluginV3OneBuild, and IPluginV3OneRuntime on one class.

Registration is a two-lane system. Lane one: REGISTER_TENSORRT_PLUGIN(Creator) (NvInferRuntime.h, L5271) is a static-initializer macro that self-registers a creator when the plugin library loads — see it live at skipLayerNormPlugin.cpp, L41 for the fused Skip-LayerNorm V3 plugin (skipLayerNormPlugin.h, L46). Lane two: the monolithic bundled library calls initLibNvInferPlugins (inferPlugin.cpp, L171), which pushes dozens of creators — ROIAlign, EfficientNMS, InstanceNormalization, GroupNorm — into the PluginCreatorRegistry (inferPlugin.cpp, L152) through registerCreator (NvInferRuntimeCommon.h, L158). Both lanes end in the same registry the builder consults mid-compile.

The tree also documents history. GroupNormalizationPlugin (groupNormalizationPlugin.h, L37) still implements the older IPluginV2DynamicExt shape, with the familiar configurePlugin / enqueue rhythm and the actual CUDA launch at groupNormalizationPlugin.cpp, L127. If you write a new plugin today, target V3 — the V2 surface is kept for compatibility and is not recommended for new work. Note that most of these plugin kernels are plain CUDA in this repo (dive into plugin/common/kernels/ for the shared math); they are an underrated tutorial on writing production tensor kernels, complementary to the template-library style of CUTLASS.

Runtime: Engines, Contexts, and Streams

The runtime side is deliberately tiny. createInferRuntime hands you an IRuntime; its one big method is deserializeCudaEngine. An ICudaEngine is immutable — you can even re-serialize it with serialize (NvInferRuntime.h, L3201) or query which tactic sources actually made it into the compiled plan via getTacticSources (NvInferRuntime.h, L3651). All mutable state lives in IExecutionContext: input shape binding, address binding, and stream-ordered launch. One engine, many contexts, is the canonical serving pattern — each worker thread gets its own context over the shared weights.

Two runtime details deserve a nod. First, hardware targeting: IRuntime::setDLACore (NvInferRuntime.h, L1919) and the builder’s getNbDLACores (NvInfer.h, L11603) expose NVIDIA’s on-SoC deep-learning accelerator as just another backend the same engine format covers. Second, plugin lifetimes at runtime: the deserializer resolves plugin creators through getPluginRegistry (NvInferRuntime.h, L5207), so any engine containing custom ops needs those plugins loaded before deserialization — a classic deployment footgun that the registry design makes visible rather than silent.

Honest Limits

The trade-offs are real and worth naming before you adopt TensorRT for a project:

• The open source is not the whole compiler. This repository ships headers, the plugin library, and the parser. The tactic-search heuristics, the fusers, and the kernel library itself live in the closed libnvinfer binary. You can read the contract but not the optimizer — the inverse of projects like OpenVINO or ONNX Runtime, where the graph transformations are all in-tree.
• Build latency is the price of throughput. Tactic timing plus JIT convolution sources can turn an engine build into minutes-long, GPU-pegged work. Timing caches help, but a cache is only valid for the exact TensorRT version (the header says so verbatim), and every version upgrade re-opens the search.
• Engines are version-locked and hardware-specific. A plan built on one GPU generation will not deserialize on another, and forward compatibility across TensorRT releases is not a design goal. That makes engine artifacts closer to compiled binaries than to portable models — plan your artifact storage accordingly.
• NVIDIA-only, full stop. No AMD, no Apple, no CPU fallback path. If your deployment matrix is heterogeneous, you will maintain a second serving chain for non-CUDA targets.
• ONNX coverage has holes. Unsupported ops throw you into plugin territory, which means writing and maintaining CUDA for exactly the layers that made you reach for TensorRT in the first place.

None of these caveats change the punchline: for fixed models on NVIDIA GPUs, the compiler’s willingness to measure before committing is why TensorRT-compiled engines routinely beat interpreted execution by healthy margins. The source tour above should give you the map to go from “it’s fast” to knowing precisely which interface is responsible for the speed — and where to look when something is not.

Enjoyed this post? Never miss out on future posts by following us