Our first tour of TensorRT-LLM followed the classic question: how does a request become tokens? We walked the configuration contract in LlmArgs, the two engine paths, the C++ batch manager with its KV cache arithmetic, and the serving tier on top. This visit asks a different question: what does one LLM(...) constructor call actually become on your operating system? The answer is a lattice of cooperating processes — a proxy that launches workers over MPI, zero-copy socket lanes that carry requests one way and responses the other, an RPC bridge that lets the frontend poke rank zero directly, a monitor that watches worker PIDs with pidfd, and inside each worker a single PyExecutor event loop threading its way through CUDA streams. The kernels get all the attention, but this lattice is what keeps a multi-GPU server alive when a worker dies mid-batch, and it is some of the most instructive process-engineering in any open inference stack. As before, this is an educational walk through the published code of NVIDIA/TensorRT-LLM on the main branch, under the Apache-2.0 license; the earlier post at /tensorrt-llm/ covers the engines and the batch manager.

The surprising headline is how much of the old C++ executor has dissolved into Python. On main, the worker’s setup_engine accepts exactly one backend — raise a ValueError for anything but pytorch — and builds a PyExecutor directly. The C++ side remains very real, but it now lives at the perimeter: request and response types exposed through nanobind, a wire format for cache state, and a transmission stack for shipping KV blocks between machines. The orchestration story, meanwhile, is pure Python, and all the better to read for it.

The executor lattice: the frontend facade hands requests to a proxy that owns zmq lanes and MPI-spawned workers, a pidfd monitor watches worker health, postprocessing runs in its own process, each worker hosts a BaseWorker around one PyExecutor whose request queue feeds the PyTorch model engine, and a native perimeter of nanobind bindings and cache transmission connects it all.

Reading the overview from left to right:

• The abstract facade at tensorrt_llm/executor/executor.py defines submit() — every flavor of executor, local or distributed, inherits from this one class.
• The GenerationExecutorProxy is the launcher process: it creates the MPI session, wires the queues in _setup_queues, spawns workers through mpi_session.submit, and dispatches results in dispatch_result_task.
• The transport lives in tensorrt_llm/executor/ipc.py: a ZeroMqQueue with PUSH, PULL, PAIR, and ROUTER sockets, plus FusedIpcQueue that batches small messages on a sender thread.
• Worker processes are GenerationExecutorWorker instances built on BaseWorker, which owns the engine and the request serialization.
• Inside each worker, PyExecutor runs the event loop, with an admission-gated inbox at ExecutorRequestQueue and PyTorchModelEngine running the scheduled batch.
• The native perimeter starts at the nanobind bindings and reaches the cache transmission agents that move KV blocks between nodes.

One object, many operating-system processes

Follow the proxy constructor and count the processes it can create. First it decides what kind of MPI session to use: a pool session that spawns fresh workers, a comm session that binds to externally launched processes, or a remote client when the environment variable for the standalone launcher is set — three deployment shapes behind one interface, validated so an external session must match the model’s world size exactly. Then _start_executor_workers pickles the worker arguments — deliberately stripping the tokenizer, whose dynamic remote-code classes cannot survive deserialization on remote nodes — and submits a worker_main task per rank. Each rank waits behind a barrier in RpcWorker.main_task: non-leader ranks set up their engine immediately, while rank zero binds an RPCServer with a thirty-two thread pool and registers the whole worker instance so any public method becomes a remote call. The proxy waits for a ready signal on a ROUTER socket, ACKs it, and only then attaches the fast-death callbacks — a deliberate ordering so a worker that dies during initialization takes the initialization-failure path rather than the engine-dead path.

The submit path is where the lattice earns its keep. The proxy’s submit checks a sticky engine-dead flag, assigns a client id, registers a fresh GenerationResult, and pushes the request onto a zmq PUSH socket. That order matters, and the code says why in a comment: if the engine dies between the dead-flag check and the result registration, a request could hang forever on a queue whose producer is gone, so the registration is followed by a second dead-check that fails the request fast. Inside the worker, BaseWorker’s submit refuses to run on any rank but zero — a guard that exists because everyone eventually forgets the if __name__ == '__main__' dance that MPI launching requires.

The queues that carry the words

The transport layer is a small zoo of sockets with a big idea behind it: requests flow one way, results flow another, and control traffic never mixes with either. _setup_queues binds a request queue the workers connect to, a result queue in PULL mode when postprocessing runs out-of-process so multiple senders can push concurrently, a ROUTER queue for per-worker initialization status, and lazily — to avoid perturbing startup timing — a PAIR ack channel for profile control calls. Multi-frontend serving extends the same pattern: the launcher creates a shared IPC directory and an HMAC key, rank zero binds one request ingress that every frontend PUSH-connects to, and each frontend owns its own result lane, all exposed through multi_frontend_attach_info for a serve process to hand out.

Results come back through a dispatch thread, not the request path. dispatch_result_task pops fused messages, routes each response to the queue of the client that owns it, and pops the result from the pending table when a response is final. Fusing happens in FusedIpcQueue: its sender thread drains up to a hundred queued objects into one zmq message, which matters when a decode step emits thousands of tiny streaming responses per second. Optionally the last hop moves to a separate process entirely: PostprocWorker owns a tokenizer and user hooks, pulls raw responses over its own pipe, and pushes detokenized Output records back toward the frontend — keeping GIL-heavy text work off the engine’s dispatch thread. The GenerationResult side of this traffic is a single response-folding method that understands engine responses, wrapped responses, and postprocessor outputs as three dialects of the same stream.

Death is a first-class event

The most sobering part of any distributed server is failure, and this codebase treats worker death as a designed state, not an exception. Liveness has two layers. The event-driven layer hangs on MPI futures: when a worker future completes, the done-callback immediately marks the engine dead and broadcasts EngineDeadError to every pending request, so propagation does not wait for the next poll tick. The polling layer catches what futures miss: WorkerProcessMonitor registers a pidfd per local worker — validated against the process start time read from procfs so a recycled PID cannot impersonate a live worker — and find_dead_worker reports exits the moment the kernel does. A background error monitor loop drains the error queue, checks remote-session death notifications, and drives pre_shutdown; remote launcher deployments get the same guarantees through a forwarded-death check at _check_remote_worker_death. Once dead means dead forever: the sticky flag turns every subsequent submit into an immediate EngineDeadError instead of a silent hang.

The same discipline extends to deliberate pauses. BaseWorker exposes sleep and wakeup with tag-validated memory transitions, and the engine’s request queue gates admission behind an explicit state machine — _enqueue_impl refuses new work while the executor is mid-sleep-transition, so you cannot enqueue a request into an engine that is releasing its cache.

Inside the worker: one PyExecutor

What actually runs in a worker process? PyExecutor is assembled from four collaborators — a resource manager, a request scheduler, a model engine, and a sampler — plus optional drafter, guided decoder, and KV transceiver. Its start_worker is a tour of threading care: ranks split across stages duplicate their MPI communicator on the main thread before any worker thread starts, because MPI_Comm_dup is collective and racing it against point-to-point traffic deadlocks; a broadcast thread relays sample state between ranks; and only then does the event loop thread start spinning. Requests enter through enqueue_requests, land in the admission-gated inbox, and responses leave through await_responses, which the worker’s AwaitResponseHelper pumps continuously and routes either to per-client queues or, in the parallel-postprocessing deployment, to the fused IPC batch lane. Each iteration ends at the model engine’s forward, which turns a scheduled batch into tensor inputs and hands them to the runner under the warmup flag during memory profiling.

The worker also hosts the small conveniences that make a long-lived server manageable. collective_rpc gives the frontend a uniform way to call into rank zero — including the multi-rank sleep and wakeup pairs, which coordinate through their own duplicated communicator and listener threads.

The detail view: the frontend of facade and result folding; the control plane of proxy, frontend proxy, monitor, and RPC client; the transport of zmq queues, message fusion, and the postprocessing worker; the worker tier of RPC server, BaseWorker, and the response pump; the PyTorch engine of request queue, event loop, and model engine; and the native perimeter of nanobind, serialization, transfer agents, UCX connections, and the CUDA buffer layer.

Walking the detail diagram, a single request crosses every tier. The facade’s submit hands the request to the proxy; the proxy’s PUSH socket delivers it to the rank-zero worker, where BaseWorker builds a tllm.Request through the nanobind bindings — the constructor at request.cpp exposes SamplingConfig and friends, and a small optimization hands the C++ side an int32 token buffer instead of a Python list when one is available. The PyExecutor inbox admits it, the scheduler schedules it, the model engine runs it, and responses flow back through the await helper, the fused result lane, the dispatch thread, and finally the result future the caller is holding. The native tier is not decorative: serialization.cpp defines the exact wire format for cache state exchanged between nodes, and when disaggregation is on, the PyExecutor hands KV blocks to the transmission stack — transfer agents loaded by a dlopen-based DynLibLoader, an AgentConnection that exchanges memory descriptors and splits VMM regions between peers, or the UCX connection manager that maintains endpoints per remote cache state. Underneath it all, BufferManager allocates device and pinned memory from CUDA pools and a WorkerPool supplies fixed device-pinned threads.

From Install to Your First Model

Installation is unchanged from the first visit — pip with NVIDIA’s extra index or a pinned container, recent-datacenter-GPU required. What this tour adds is the operational view. The single-process, single-GPU case hides the lattice entirely: one process, PyExecutor’s thread family, no zmq. Ask for tensor parallelism and the proxy spawns a worker per rank over a pool session; set the standalone launcher environment and the workers attach from outside; raise the number of serve frontends and extra processes connect to shared ingress with HMAC-signed lanes. trtllm-serve builds on the same executor, which is why its process tree mirrors the diagram above. The knobs worth knowing from this layer: profiling is a control request that travels the profile ack queue with a synchronous wait; stats and KV-cache events are fetched via the RPC bridge rather than broadcast; and health is a real endpoint backed by the monitor, not a ping handler pretending to check anything.

Honest limits: the process lattice is engineered for NVIDIA’s MPI-and-CUDA world, and while the orchestration code itself is portable Python, the meaningful paths assume Linux, pidfd, UCX or agent transports, and recent GPUs — none of this story unfolds on a laptop the way it does on a four-GPU box. The dual-backend era is over on main, so readers of older documentation will find the TRT-engine path gone and the PyExecutor path authoritative. And the code moves fast: the very structure described here — proxy, RPC, monitor — did not exist in this shape a few releases ago, which is precisely why it is worth reading now. For contrast across the ecosystem, our walks of LocalAI, Xinference, and vLLM each solved the same multi-process problem with a different geometry: gRPC backends, actor clusters, and core engines respectively. TensorRT-LLM’s answer — an MPI-spawned lattice with a sticky dead state and pidfd liveness — is the most explicit of the three, and that explicitness is exactly what makes it such a good map of what production serving really is.

Enjoyed this post? Never miss out on future posts by following us