Our first walk of openai/triton followed the compiler’s front door: how a decorated Python function becomes an AST, then TTIR, then the GPU-shaped dialects, then PTX or GCN. That leaves the other half of the story untouched — the part that runs every single frame, not just once per shape. When vLLM or a torch.compile’d model calls kernel[grid](args) for the ten-thousandth time, no compilation happens at all. What happens instead is one of the most carefully tuned hot paths in the Python ecosystem: a generated binder re-packs the arguments, a two-level cache turns the call into a key, a launch-metadata lazily assembles only if a profiler is watching, and a per-target launcher finally hands the whole thing to a small C extension that speaks the CUDA driver API directly. This tour follows that path end to end, from JITFunction.run in Python to cuLaunchKernelEx in C, with a stop in the NVIDIA backend to see how much of the compile decision is already settled by compute capability. As always, this is an educational read of published source on the main branch, under the MIT license; the earlier post at /triton/ covers the frontend and the MLIR dialects.

The headline is how much of the design is organized around one fact: on a healthy system, the cache-hit branch of run() must be nearly free. Everything else — the binder generated at import time, metadata packed once into a tuple, handles loaded lazily exactly once, hooks that short-circuit to None when no profiler is attached — serves that goal.

The launch path: JITFunction.run consults the per-device binder and kernel cache, a miss goes through CompiledKernel into the CUDABackend's stage sequence and the C++ MLIR lowering, while a hit goes straight to the launcher created in _init_handles — the CudaLauncher that forwards to driver.c and cuLaunchKernelEx.

Reading the overview from left to right:

• JITFunction.run is the hot entry: binder call, cache key, launch. The binder itself is generated from the signature, and one exists per device in device_caches.
• FileCacheManager persists compile artifacts in hash-keyed directories.
• CompiledKernel holds the per-stage artifacts and loads CUDA module and function handles lazily in _init_handles.
• CUDABackend owns the stage sequence and every capability-gated option.
• The C++ side lowers TritonGPU IR in ConvertTritonGPUToLLVM, with shared-memory allocation computed per layout conversion.
• CudaLauncher builds argument metadata and forwards everything to the launch function in driver.c.

The hit path: one binder, two caches, three guards

run() starts by fetching the current device and stream — two driver calls that later decide which cache to use. Then comes the binder: a function created once by create_function_from_signature that binds arguments, classifies constants, and produces the specialization tuple in a single pass of straight Python. Its output feeds compute_cache_key, and the key is looked up in the kernel cache. Only a miss reaches _do_compile and the _pack_args path where backend options are parsed. This is the quiet half of Triton’s speed: the expensive work of understanding a Python signature happens once per device, not once per launch.

Even the cache-hit branch runs guards, and each one exists because someone got burned. Lines 773 through 776 re-read every global variable the kernel captured and raise if one changed — without this, editing a constant in a notebook would silently keep using the old value baked into the binary. The grid canonicalization accepts callables so a kernel can compute its own launch size from the bound arguments, defaulting missing dims to 1. And the final launch at lines 788 through 789 passes launch_metadata — built by a method that returns None immediately when no launch hook is installed, so the profiling path costs nothing when nobody is listening.

CompiledKernel: lazy handles and one launcher per source

The object the cache stores is deliberately inert. Its constructor reads the metadata JSON, restores the target from a dict into a GPUTarget, packs the metadata once via the backend’s pack_metadata — on NVIDIA that is just the tuple (num_warps, num_ctas, shared) — and loads every compilation stage’s text into an AsmDict: ttir, ttgir, llir, ptx, and the cubin bytes. What it deliberately does not do is touch the driver. module, function, and _run start as None, and only the run property triggers _init_handles, which instantiates the backend launcher class, checks the kernel’s shared-memory request against the device maximum, checks tensor-memory limits on Blackwell, and finally calls load_binary to get a CUfunction plus register and spill counts. Even failure is engineered: the raise_ helper deep-copies the exception into a closure so a cached kernel that always fails does not keep its whole traceback alive in the global cache.

There is a second, lower-level entry for people who want to launch without the JIT machinery at all: CompiledKernel.getitem returns a plain runner closure taking raw grid and stream — the escape hatch that serving frameworks use when they want to manage streams themselves.

Capability decides: CUDABackend’s stage sequence

The NVIDIA backend opens with an opinion worth quoting in spirit: its constructor stores the numeric compute capability and sets the binary extension to cubin. Everything downstream branches on that integer. parse_options turns sanitizers on debug mode, forces max_occupancy=1 under the gsan instrumentation, refuses the clc option below SM100, refuses multi-CTA below SM90, extends the supported fp8 set at capability 89 and above, and defaults Hopper’s imprecise-accumulator ceiling. The make_ttgir sequence reads like a generation table: after the convert-to-TTGIR pass, Ampere gets a prefetch pass at line 358; the Hopper branch at lines 328 through 337 adds warp specialization with latency assignment and loop scheduling for software pipelining; the Blackwell branch at lines 338 through 353 hoists tensor-memory allocations, promotes operands to tmem, and runs warp specialization again over partitioned warps. Common tail: TMA lowering at line 364, fence insertion at line 374, MMA lowering at line 375.

The back half of compilation leaves MLIR entirely. make_ptx calls LLVM’s translate-to-asm, then regex-finds the single .visible .entry name for the metadata, rewrites the .version and .target lines to the chosen PTX and SM, and strips the debug flag that would stop ptxas from optimizing. make_cubin writes the PTX to a temp file and shells out to ptxas — a real external binary resolved per architecture — with line-info only by default, --fmad=false when fp fusion is disabled, and a pinned register-allocation level that works around a ptxas 13.x bug. It is a sobering reminder that this “Python compiler” ends with string rewriting and a subprocess.

The detail view: the runtime tier of JITFunction, caches, knobs, and the user allocator; the compile tier of CompiledKernel and the AST code generator; the backend of CUDABackend, CudaLauncher, and driver.c; the C++ lowering of the LLVM conversion pass and its pattern families; and the allocation and barrier passes underneath.

Walking the detail diagram, the compile leg crosses the C++ boundary at ConvertTritonGPUToLLVM, whose populateConversionPatterns wires in the NVIDIA pattern families: layout conversions with swizzling, tensor-map creation and the fenceproxy-acquire that TMA descriptors require, fp8 and fp16 cast sequences for elementwise ops, and the heavyweight LoadStoreOpToLLVM, whose atomic-RMW conversion even promotes atomics to plain load-acquire when the access pattern allows it. Before any of that runs, the AllocateSharedMemoryNv pass has already decided which conversions need scratch space and where each byte sits. On the runtime side, global and profile scratch buffers come from the user allocator registered with set_allocator — a deliberately tiny protocol, with a null allocator that raises a helpful message when a kernel needs scratch and nobody registered one.

The last hop: CudaLauncher and the C extension

The launcher is built per compiled kernel. CudaLauncher.init decides between the TMA argument ABI and a decomposed one based on whether tensor descriptors appear in the metadata, injects three extra signature entries when the gsan instrumentation is on, and — quietly one of the neatest tricks in the file — computes a shared-memory budget: when a kernel was compiled with a max_occupancy request, lines 267 through 279 round the per-CTA budget down to 128-byte units and subtract the driver’s reserved shared memory, so compilation stays device-independent while launch-time reservation does not. It then builds the argument metadata once through the C extension and wraps the extension’s launch function, optionally with a tensordesc wrapper.

The call allocates global scratch through the user allocator and profile scratch through either the profiler’s allocator or a default one, refuses to run gsan kernels under CUDA graph capture, and re-packs the metadata tuple with the resolved shared size before crossing into C. The C side, driver.c, loaded the cubin with cuModuleLoadData back in _init_handles; at launch it builds a CUlaunchConfig whose grid X is multiplied by the CTA count and whose block X is 32 times the warp count, then attaches up to four launch attributes as needed: an oversized-shared-memory mode above 228 KB, programmatic dependent launch, cooperative launch, and cluster dimensions plus a spread scheduling policy, with a non-portable allowance when sixteen CTAs per cluster are requested. One call to cuLaunchKernelEx at line 1075 and the kernel is running.

From Install to Your First Model

After pip install triton, nothing above needs configuration — but the knobs make the layers visible. knobs.runtime.launch_enter_hook and its exit twin are what profilers attach to, and the whole launch_metadata machinery exists so those hooks can see grid, stream, and arguments without paying for it otherwise; TRITON_INTERPRET=1 swaps the compiled path for the pure-Python interpreter covered in the first tour. The observable side effects of this tour are in CompiledKernel objects themselves: n_regs and n_spills from ptxas, metadata.shared from the allocation pass, and the per-stage IR strings in asm — the same artifacts the disk cache persists.

Honest limits: this walk deliberately stayed on the NVIDIA path — the AMD backend mirrors the same stage structure with its own GCN-lowering tree, and the interpreter and Gluon paths diverge from the launcher story entirely. We read code, not benchmarks; the “nearly free” claim about the hit path is structural, not measured. And the launch surface moves fast — attribute handling, tensor descriptors, and the scratch protocol all grew within the last year, which is exactly why this layer rewards a second reading. For context in the ecosystem: vLLM and SGLang spend their days inside the hot path described here, TensorRT-LLM replaces the launcher with a full process lattice, and OpenVINO answers the same “compile once, launch cheaply” question on CPU. Triton’s answer — a generated binder, a tuple of packed metadata, and four launch attributes away from the driver — is the leanest of the lot.

Enjoyed this post? Never miss out on future posts by following us