Between 2019 and 2022, if you wanted transformer inference to actually go fast on NVIDIA GPUs, you ended up reading FasterTransformer. It was NVIDIA’s C++/CUDA laboratory for the transformer block: attention, FFN, layernorm, sampling, and weight quantization treated as kernel-engineering problems rather than framework conveniences. It shipped encoder paths for BERT, decoder and decoding paths for machine translation, and a full GPT serving stack with bindings for TensorFlow, PyTorch, and a Triton backend. One honest note before anything else: the repository is frozen. The very first line of its README states that development has transitioned to TensorRT-LLM and that the repo “will not have further development.” Everything below should be read as a blueprint from that archive, not as a living project.

Why spend a post on frozen code? Because the blueprints escaped. The decode-time multi-head attention kernel we will visit is a direct ancestor of the one vLLM launched with, and its custom allreduce idea reappeared in TensorRT-LLM and vLLM almost unchanged in spirit. Its weight-only-quantization GEMM scaffolding, built on CUTLASS (a codebase we toured separately here), foreshadowed the entire W4A16 wave that ktransformers-style hybrid inference rides today. Reading FasterTransformer is reading the lab notebook that modern serving systems copied from.

This post walks the GPT serving path from a Triton model config down to a single CUDA kernel launch, using two diagrams generated directly from the src/fastertransformer tree on branch main: a compact overview first, and a detailed 34-node map later in the post. Every line anchor we link was verified against the working tree. If you want a warm-up on why token-level serving matters at all, our inference overview and the Triton Inference Server tour are good companions.

Overview of the FasterTransformer GPT serving stack: a Triton model builder instantiates the ParallelGpt engine, which drives context and step decoders over tensor-parallel attention and FFN layers, backed by fused kernels, custom allreduce, and a cuBLAS wrapper.

Reading the overview from left to right:

• ParallelGptTritonModel, L306 — the serving entry point; it parses a Triton config.ini, weights, and MPI world size, then spawns one engine per GPU and hands out model instances.
• ParallelGpt::forward, L534 — the top-level engine loop: prefill over the prompt, then one step per generated token, sampling, stopping, and tensor-parallel logit gathers.
• ParallelGptContextDecoder::forward, L303 — the prefill pass, which pushes all prompt tokens through the block stack at once and fills the KV cache.
• ParallelGptDecoder::forward, L257 — the incremental step decoder, invoked once per token at L1346 of ParallelGpt.cc.
• TensorParallelGptContextAttentionLayer::forward, L23 and TensorParallelDecoderSelfAttentionLayer::forward, L185 — thin wrappers that shard attention GEMMs across ranks and allreduce the partial outputs.
• TensorParallelGeluFfnLayer::forward, L33 — the column-parallel/row-parallel FFN sandwich around FfnLayer::forward, L34.
• masked_multihead_attention, L72 — the famous single-kernel decode attention, called from DecoderSelfAttentionLayer.cc, L144.
• oneShotAllReduceKernel, L139 — the custom allreduce over P2P-mapped peer buffers, with NCCL (ftNcclAllReduceSum, L56) as the fallback.
• invokeTopKSampling, L439 — curand-seeded top-k/top-p token selection at the end of each step.

The detailed map: 34 nodes from Triton instances and the PyTorch op through the GPT engine, tensor-parallel layers, and the kernel layer — the same structure the rest of this post follows.

A Tensor Class That Carries Its Own Memory Type

The root abstraction is small. struct Tensor at L107 is a pointer plus a data type, a shape, and a memory-type tag (CPU, GPU, or framework-owned), with helpers to convert to and from PyTorch and TensorFlow tensors. A TensorMap is then just a string-keyed bag of tensors, and it becomes the calling convention for every layer: layers read inputs and write outputs by name. That is the entire interface glue, and it is why the same C++ core can be driven by three different frameworks.

Below the tensors sits an allocator hierarchy. IAllocator at L64 has specializations for raw CUDA allocation, PyTorch’s allocator, and TensorFlow’s BTF allocator, all exposing the same malloc/free/reMalloc surface with a shared pooling idea: freed device buffers are cached by size class rather than returned to CUDA immediately. Every layer inherits a buffer-lifecycle contract from BaseLayer’s allocateBuffer at L54, and small utilities like deviceMalloc at L29 fill the gaps. This design is invisible from Python but explains a lot of the speed: there is no hidden host-device synchronization at layer boundaries, because the memory contract is explicit.

The PyTorch side of the bridge is GptOp::forward at L106, a custom op that converts torch::Tensor inputs into ft::Tensor, calls the engine, and converts outputs back. If you have read our PyTorch extension material, this is the heavyweight industrial version of the same pattern.

The Decode Kernel Everyone Copied

GPT generation is memory-bound: for each new token, the model reads every weight once and performs little arithmetic. FasterTransformer’s most influential attack on that problem was to collapse an entire incremental attention step — reading the QKV projection, adding bias, applying rotary-free positional state, computing softmax against the cached keys, and appending the new K/V to the cache — into a single CUDA kernel per layer. The launcher is masked_multihead_attention at L72, and the actual GPU code is a template, masked_multihead_attention_kernel at L1119, instantiated per head dimension in sibling files sized from 32 up to 256.

The kernel keeps a query vector in registers, streams over cached keys and values in shared-memory tiles, and writes the updated cache slot — no intermediate global-memory round trip for attention scores. This is the kernel that vLLM’s early versions shipped a port of, and its one-warp-per-head layout is recognizable in several successor projects. It is also the reason the file tree looks the way it does: a directory of per-head-size instantiations generated from one template, so nvcc can compile each shape at its best register pressure.

The prefill (context) path is a different beast because sequence lengths are large and compute-bound, so it runs a more conventional multi-stage flow: GptContextAttentionLayer::forward at L25 runs one fused QKV GEMM through the cuBLAS wrapper, then invokeAddFusedQKVBiasTranspose at L1502 applies bias and splits heads in a single pass, and a masked softmax follows. For encoder models the repo goes further and calls into TensorRT’s fused flash-attention cubins under 3rdparty/trt_fused_multihead_attention — the same family of techniques our FlashAttention tour covers from the research side.

Tensor Parallelism Without the Tax

The multi-GPU GPT lives under models/multi_gpu_gpt, and its structure is a case study in how little code tensor parallelism actually needs if the layers cooperate. The block-stack constructors wire themselves up in a few lines — ParallelGptDecoder.cc at L27 creates a TensorParallelDecoderSelfAttentionLayer and at L44 a TensorParallelGeluFfnLayer — and from there every GEMM is head-sharded or column-sharded, followed by an allreduce of partial sums.

The interesting part is the allreduce backend. The default is NCCL, initialized in ftNcclInitialize at L308 and invoked through ftNcclAllReduceSum at L56; the engine also allgathers final logits across the tensor-parallel group so every rank can sample the same next token. But NCCL’s per-call overhead hurts at decode-time batch sizes, so FasterTransformer added its own allreduce: peer GPUs map each other’s device memory with P2P APIs, CustomAllReduceComm::customAllReduce at L46 negotiates the peer pointers and swaps in an internal buffer, and then either oneShotAllReduceKernel at L139 or twoShotAllReduceKernel at L202 finishes the reduction in a single kernel launch. DeepSpeed later popularized a near-identical mechanism, which we looked at in the DeepSpeed tour.

Sampling and stopping are their own kernel zoo: invokeTopPSampling at L1208 for nucleus sampling, beam_online_softmax_topk_kernel at L297 for streaming beam search, invokeGatherTree at L636 to backtrace finished beams into sequences, and invokeLengthCriterion at L135 to mark EOS and max-length stops on-device.

From Install to Your First Model

The documented path in docs/gpt_guide.md is container-first, and that is still the most reliable way to build it. Start from a recent NGC PyTorch container (the guide uses nvcr.io/nvidia/pytorch:22.09-py3-era images), clone the repository, and pull its submodules — git submodule init && git submodule update — since the CUTLASS-based kernels and the TensorRT fused-attention assets live in 3rdparty. Then build with CMake, selecting your GPU’s compute capability:

mkdir build && cd build
cmake -DSM=80 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_MULTI_GPU=ON ..
make -j12

-DBUILD_PYT=ON produces the PyTorch custom op; -DBUILD_MULTI_GPU=ON enables the tensor-parallel path; dropping -DBUILD_PYT and adding the TensorFlow flag builds that binding instead. Before running anything, do what every FasterTransformer user ritualistically did: run the GEMM sweep, generate_gpt_gemm_config at L27, which benchmarks cuBLAS candidates across your shapes and writes the winners into gemm_config.ini — for example ./bin/gpt_gemm 8 1 32 12 128 6144 51200 1 1 0 for a GPT-2-XL-shaped sweep. At runtime cublasMMWrapper::Gemm at L94 consults that cache per shape, which is why a model without a tuned config can be noticeably slower.

For a first end-to-end run, convert a Megatron- or HuggingFace-format GPT-2 checkpoint into FT’s split format, point examples/cpp/multi_gpu_gpt/gpt_config.ini at it (that file controls model path, tensor-parallel size, and decoding hyper-parameters), and run ./bin/multi_gpu_gpt_example on one GPU or mpirun -n 8 ./bin/multi_gpu_gpt_example across eight. The PyTorch route is the same idea through the op: python ../examples/pytorch/gpt/multi_gpu_gpt_example.py --tensor_para_size=8 --ckpt_path=.... For actual serving, the Triton backend builds into a shared library you drop into a Triton model repository; the model config maps one-to-one onto what ParallelGptTritonModel reads. Our Triton Inference Server overview explains the server side of that contract, and TGI represents where that serving layer went next.

Weights, LayerNorm, and the Plumbing Nobody Photographs

Two supporting cast members deserve a mention. Weight loading is split cleanly: ParallelGptWeight::loadModel at L293 reads embeddings and the output kernel, while ParallelGptDecoderLayerWeight::loadModel at L275 reads each layer’s QKV, FFN, and layernorm tensors, with INT8 and FP8 variants handled by sibling weight classes. Around the two big GEMMs of every block, small fused kernels do the glue work on-device: invokeGeneralLayerNorm at L1653 (RMSNorm included, which is how the repo supported NeoX-style variants), invokeGenericActivation at L243 fusing bias and GELU/Silu into the FFN middle product, and invokeAddBiasResidual at L55 for the residual adds. Embedding lookup with positional encoding and prompt-past splicing lives in gpt_kernels.cu at L120. None of these is glamorous; together they are the reason a GPT step touches DRAM once per tensor instead of three times. For contrast, nanoGPT and llm.c are excellent minimal implementations that make FT’s engineering density obvious.

Honest Limits

Read the archive banner first: the README’s opening line says FasterTransformer transitioned to TensorRT-LLM and will receive no further development. The last commits date to 2023-era dependencies, and the FP8 work targets Hopper-era CUDA 11.8 images. Nothing here tracks newer architectures, MLA-style attention, or speculative decoding.

Build complexity is real. Submodule pulls, per-architecture SM flags, and template instantiation per dtype and head dimension make for long compiles and large binaries. The C++ API is heavy for the same reasons the kernels are fast: you assemble TensorMaps by hand, size activation buffers through the allocator, configure gemm_config.ini per shape, and wire MPI yourself. There is no runtime kernel autotuning of the kind modern SGLang-style stacks take for granted. Quantization is powerful but hardware-gated — INT8 paths need Turing or newer, FP8 needs Hopper — and the support matrix in the README is explicit about which framework binding gets which feature.

Finally, the world moved on. TensorRT-LLM is NVIDIA’s sanctioned successor, and the kernel ideas survive there and in vLLM; CTranslate2, TGI, and SGLang cover the practical serving space with far less ceremony. But as a map of how far careful kernel engineering pushed transformer inference before the modern frameworks existed, FasterTransformer remains one of the most instructive trees you can clone — a frozen lab, and a blueprint everyone’s serving stack still carries traces of.

Enjoyed this post? Never miss out on future posts by following us