Almost every framework we have toured on this blog sits on the same substrate. vLLM [(/vllm-2/)] spends its effort on KV-cache management and batch scheduling, but its tensor math is PyTorch. TorchServe [(/torchserve/)] and ExecuTorch [(/executorch/)] exist to package and run models that were defined and trained in PyTorch in the first place. Even compact educational engines like nanoGPT [(/nanogpt/)] are best understood as deliberate shrinks of what PyTorch does. So it is worth opening the box itself: the pytorch/pytorch repository is one of the most instructive C++ codebases in open source, and its layering explains much of the behavior you observe from Python every day.

This tour follows a single op call from the very top to the very bottom. You write y = model(x) and loss.backward() in Python; we will end at a CUDA kernel writing its results into memory that was carved out of a cached block pool by the caching allocator. Between those two points sit five distinct layers: the Python module machinery, the operator registry, the C++ dispatcher, the autograd engine, and the backend kernels with their memory management. Each layer has a clean job, and each one is independently interesting.

A note on method: every file link below points at a specific line in the pytorch/pytorch repository on the main branch, and every line number was verified against a fresh checkout while writing this. Line numbers drift on a codebase this active, but the symbols are stable; if a link lands a line or two off, search for the symbol name in the same file.

The overview map: a Python nn.Module call descends through the dispatcher into the autograd engine, backend kernels, and finally the CUDA caching allocator, while the optimizer closes the loop back to the weights.

Reading the overview from left to right:

• _wrapped_call_impl, L1783 and _call_impl, L1791: the Python entry point where forward hooks run before your own forward method is invoked.
• Parameter, L30: a torch.Tensor subclass whose requires_grad flag defaults to True, so every weight a module owns is automatically tracked.
• native_functions.yaml, L1: the declarative registry where every aten operator is named and typed before code generation turns it into C++ and Python bindings.
• Dispatcher::findSchemaOrThrow, L150 and TypedOperatorHandle::call, L623: how a call by name becomes a typed C++ fast path.
• TensorImpl.key_set_, L650 and DispatchKeySet, L167: the bitset on every tensor that decides whether a call goes to autograd first and then to a concrete backend.
• autograd Node, L130 and Engine::execute, L1428: the recorded graph and the engine that replays it on backward.
• CPU mul_kernel, L126 and CUDA mul_kernel_cuda, L22: the same operator, two concrete kernels behind one dispatch table.
• CUDACachingAllocator malloc, L1846: where the output tensor’s GPU storage actually comes from.

One Call, Five Layers

Take the simplest possible training step: a linear layer applied to a batch, then a loss. On the Python side, model(x) does not call your forward method directly. The class attribute __call__ is bound to _wrapped_call_impl at module.py L1926, which forwards into _call_impl at L1791. That is where hooks live: global and per-module forward hooks are wrapped so they can see the owning module, a job done by _WrappedHook, whose call, L85 injects the module reference before delegating to the user’s hook. Only after the hook machinery does your forward run, and for a linear layer it ends in F.linear, which calls the C++ operator aten::addmm.

Now the C++ side takes over, and the fast path is worth reading slowly. The typed template Dispatcher::call begins at Dispatcher.h L784: it asks the handle’s dispatchKeyExtractor, L505 to compute a key set from the tensor arguments, then looks up the matching kernel and invokes it. If the arguments require gradient, the autograd key is in the set, so the call first lands in an autograd kernel, which records what happened and then redispatches to the real backend kernel. The result flows back up to Python as an ordinary tensor that quietly carries a gradient function with it.

The detail map: the dispatcher's typed call path, the tensor-side key set, the engine's queue-and-drain loop, and the two backend kernels that share a TensorIterator-style interface.

Inside the Dispatcher

The dispatcher is the reason PyTorch can support autograd, tracing, functional transforms, and half a dozen hardware backends without turning every operator function into spaghetti. The design starts from native_functions.yaml, L1, a declarative file listing every operator’s name, signature, and where its implementations live. Code generation reads it and produces the bindings you use from Python and C++.

At runtime each operator is represented by an OperatorEntry, L70, which holds a table mapping dispatch keys to registered kernels. The same aten::mul name can simultaneously have an autograd wrapper, a composite implementation, a CPU kernel, and a CUDA kernel registered under different keys. Choosing among them happens through key sets: DispatchKeyExtractor, L128 knows which arguments matter, and its getDispatchKeySetUnboxed, L194 ORs the key sets of all tensor arguments into one bitset. That bitset is built from BackendComponent, L65 values, an enum of CPU, CUDA, and other backends packed as bit positions, and wrapped in the DispatchKeySet, L167 class every TensorImpl carries.

This is also the extension point for the wider ecosystem. Custom CUDA libraries like FlashAttention [(/flash-attention/)] and Triton [(/triton/)] kernels can be registered as first-class operators through torch.library, at which point they ride the exact same path as built-ins. Serving frameworks like vLLM [(/vllm-2/)] use the same mechanism to register fused attention and sampling ops that the dispatcher cannot distinguish from native ones.

The Autograd Engine

Forward execution records; backward execution computes. Every differentiable op contributes an autograd Node, L130, which stores the edges to its consumers and an apply function that computes one backward step. When you call loss.backward(), the engine takes over at Engine::execute, L1428. It first runs compute_dependencies, L1360 (invoked at L1494) to count how many gradient contributions each node is waiting for, then pushes ready nodes into per-device queues. Those queues are the device_ready_queues_, L246 members of the Engine struct, L131, one worker thread per device.

From there the loop is thread_main, L575: pop a node, accumulate its incoming gradients into an InputBuffer, L41, call the node’s apply, and push whatever becomes ready. Gradients arriving through several branches are summed in the buffer before the node fires, which is exactly the semantics you want from a fork-join graph. When the last node finishes, exec_post_processing, L765 synchronizes per-device streams and runs final callbacks.

This engine is why distributed trainers can hook in cleanly. DeepSpeed [(/deepspeed/)] and Accelerate [(/accelerate/)] wrap the standard step with gradient accumulation hooks, gradient clipping, and all-reduce calls, but the underlying graph walk is the one described above. And once gradients have landed on each Parameter’s .grad field, Adam.step, L216 consumes them, expressing its elementwise updates as ordinary dispatched tensor ops, including fused foreach variants that batch many parameters into fewer kernel launches.

Kernels and Memory

Below autograd sits the code that actually multiplies numbers. Binary elementwise ops converge on TensorIteratorBase, L246, which normalizes shapes and strides, handles broadcasting, picks a common dtype, and decides the loop structure. On top of it, the CPU mul_kernel, L126 runs vectorized inner loops selected per scalar type, while the CUDA mul_kernel_cuda, L22 hands an elementwise functor to a GPU launch helper. Two files, one contract: that contract is what makes an operator portable.

Memory is the last layer and a quietly important one. Calling cudaMalloc for every activation would dominate latency, so the CUDACachingAllocator, L1846 malloc instead splits, rounds, and recycles cached blocks that are stream-aware: a block freed on one stream is only reused where it is safe. This caching behavior is why GPU memory in PyTorch looks “stuck” at a high watermark after a big batch, and why torch.cuda.empty_cache() exists as an escape hatch. Projects like CUTLASS [(/cutlass/)] plug in below this line, providing the tile-level GEMM primitives that the highest-performance kernels call into.

From Install to Your First Model

The easiest way to get all of this running is the official wheel. For a CPU-only setup, pip install torch is enough; for a CUDA build, pick the wheel index matching your toolkit:

pip install torch --index-url https://download.pytorch.org/whl/cu124

Then a complete training loop, small enough to read and large enough to touch every layer of this tour:

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

for step in range(10):
    x = torch.randn(64, 784)
    y = torch.randint(0, 10, (64,))
    pred = model(x)          # __call__ -> hooks -> forward -> aten ops
    loss = loss_fn(pred, y)  # dispatcher records an autograd graph
    opt.zero_grad()
    loss.backward()          # Engine::execute walks the recorded graph
    opt.step()               # Adam.step updates every Parameter

Each line maps to a layer from the diagrams: model(x) enters through _call_impl, the arithmetic dispatches through OperatorEntry tables into CPU or CUDA kernels, loss.backward() hands the recorded nodes to the engine and its per-device queues, and opt.step() runs the update math through the dispatcher again. If you want to appreciate how much engineering hides in those lines, try nanoGPT [(/nanogpt/)] or tinygrad [(/tinygrad/)], where the same concepts are implemented in a few thousand lines instead of a few hundred thousand.

Honest Limits

A tour like this compresses, so here is what it leaves out. Building PyTorch from source is a serious undertaking: CMake-driven, with dozens of optional dependencies, and a compile time measured in tens of minutes to hours depending on your machine. The build system and the code generators under tools/ that turn native_functions.yaml into thousands of lines of glue are a project of their own and were skipped entirely. Also skipped: torch.compile and its Inductor code generator, the functorch and vmap transforms that layer above the dispatcher, the distributed c10d stack, and the newer backends for MPS and other accelerators.

The dispatcher itself has a genuine learning curve. Dispatch keys, include and exclude sets, thread-local state, and the boxed versus unboxed calling conventions interact in ways that only make sense after a few debugging sessions; the tour presented the happy path where autograd wraps and redispatches, but real traces show fallthroughs, aliases, and special cases. The diagrams simplify deliberately: an inference-mode tensor skips autograd entirely, and a compiled graph may never touch the dispatcher for fused regions at all. Finally, remember that line numbers in a repository this active move weekly; treat every deep link here as a snapshot of a moving target, and use the symbol names as the durable handle.

Enjoyed this post? Never miss out on future posts by following us