Most inference engines we have toured — vLLM, SGLang, TensorRT-LLM — are built around a single strong idea: batching. NVIDIA’s Triton Inference Server is built around a different one: being the front door. It speaks half a dozen protocols, runs any framework through its backend system, and lets one server process host models with wildly different shapes and schedules. In recent releases the project did something bold for a flagship repository: it split the inference core out into its own repo and left the server repository as the shell — the binary, the frontends, and the glue. That split makes this codebase uniquely readable, because everything left here is about the same question: how does one process expose one engine over many doors at once?

We will follow a request from the command line to a response: bootstrap in main.cc, the C API boundary, the evhtp-based HTTP frontend, the gRPC state machines, the shared memory manager that clients use to skip serialization, the cloud-specific frontends for SageMaker and Azure, and the newer Python OpenAI-compatible frontend that turns Triton into a drop-in chat-completions endpoint — the same trick we saw in LocalAI and Text Embeddings Inference, but here layered on a general-purpose server.

Triton's shell architecture: main.cc bootstraps the C-API server, StartEndpoints wires the HTTP and gRPC frontends plus shared memory, and the Python OpenAI frontend runs as a managed companion.

Reading the overview from left to right:

• main parses options, creates the core server, and blocks on signals.
• TritonParser::Parse turns hundreds of flags into ServerOptions.
• TRITONSERVER_ServerNew crosses into libtritonserver — the engine lives in a sibling repository now.
• StartEndpoints wires the frontends you compiled in.
• HandleInfer is the HTTP v2 inference path; CommonHandler::Start registers the gRPC services.
• SharedMemoryManager tracks CUDA IPC and system-shared regions.
• TritonLLMEngine drives the OpenAI-compatible frontend.

Bootstrap: Flags, C API, Signals

The entry point in main.cc is short, and that brevity is architectural. First, TritonParser::Parse consumes argv — the flag registry in SetupOptionGroups covers everything from HTTP threads to cache configuration to per-backend settings (ParseBackendConfigOption at line 2053). Then BuildTritonServerOptions translates that blob into the C structs the core expects, and TRITONSERVER_ServerNew hands them to libtritonserver.

Everything after that point is shell. StartTracing enables OpenTelemetry-style request tracing, RegisterSignalHandler traps SIGINT/SIGTERM for graceful drain, and StartEndpoints starts whichever frontends were compiled in. The main thread then parks in a loop waiting on the signal flag — every request that will ever be served flows through endpoints started in that one call. If you compare this with ONNX Runtime, where the “server” concept barely exists, or TorchServe, which wraps a Java-style management plane around Python workers, Triton’s choice is stark: the C API is the platform, and every frontend is a client of it.

The HTTP Frontend: evhtp and the v2 Protocol

The HTTP frontend in http_server.cc is built on libevent’s evhtp. The Dispatch callback receives every request and the big URI switch (lines 4913-4964) maps it onto the KServe v2 protocol: HandleModelReady for health, HandleModelMetadata and HandleModelConfig for introspection, and HandleInfer for the real work. HandleInfer is where a JSON body becomes tensors: it parses inputs, resolves binary-vs-JSON tensor formats, builds a TRITONSERVER_InferenceRequest, and calls into the C API asynchronously — the response comes back on a callback thread, and evhtp serializes it out.

Streaming gets its own machinery: chunked transfer for generative backends, with ChunkCountIncrement policing the chunk budget. What impressed us most is how little HTTP-specific logic there is — the file is long because the v2 protocol has many endpoints, not because it hides business rules. That is the benefit of the C API split: this file orchestrates, it never computes.

The gRPC Frontend: State Machines per Request

gRPC works differently from HTTP: instead of one callback per request, Triton runs handler objects with explicit state machines. CommonHandler::Start registers each service — including metrics, model control, and streaming — and stops with drain semantics in Stop. The inference path in infer_handler.cc is a lifecycle: Process steps through request states, Execute issues the async C API call, and InferResponseComplete finishes the gRPC response when the engine calls back.

Streaming inference — essential for LLM token streams — is ModelStreamInferHandler: one bidirectional gRPC stream carries many requests, Process multiplexes responses back by request ID. The Python OpenAI frontend later leans on exactly this stream, which is why its code path matters even if you never write a gRPC client yourself.

The two frontends also differ in a subtle way that pays off under load. HTTP handles one request per connection callback, so concurrency scales with evented sockets — cheap for small requests, and the chunked streaming path covers generative models. gRPC instead keeps per-request state objects alive across the full inference duration, which lets the server accept more in-flight work than it can immediately execute; backpressure lives in the handler state machine rather than in connection limits. Both paths converge on the same async C API call, which means a model author writing a backend sees neither protocol — the frontend differences end exactly where libtritonserver begins. That separation is also why Triton can enable protocols independently at compile time: main.cc guards each frontend behind a build flag, so a minimal deployment can ship HTTP only, or gRPC only, without dead code in the binary.

Shared Memory: Skipping the Copies

For large inputs, serializing tensors through HTTP or gRPC bodies is pure overhead. Triton’s answer is a registration model: clients create a region in /dev/shm or a CUDA IPC handle, register it with the server, then reference it by name in requests. The manager tracks these in RegisterSystemSharedMemory and RegisterCUDASharedMemory, serves introspection through GetMemoryInfo, and tears regions down in Unregister. main.cc constructs the manager up front (line 509) and every frontend receives the same instance, so a CUDA region registered once is usable from any protocol. Between this and the pinned-memory tricks in faster-whisper, the lesson repeats: at inference scale, the copy is the latency.

Cloud Frontends and the OpenAI Layer

Two cloud adapters show how far the “reuse the handler” pattern stretches. SagemakerAPIServer::Create builds a frontend whose Handle unwraps SageMaker’s envelope headers and then calls the ordinary HTTP handlers (HandleInfer at line 164) — the cloud protocol is a translation layer, not a fork. The Azure-flavored orca_http.cc follows suit, formatting per-request KV-cache metrics through OrcaKVMetricHeader after ExtractKVMetrics pulls them from the core.

The newest door is Python: the OpenAI frontend is a FastAPI app (FastApiFrontend at line 50) whose create_chat_completion streams chunks through an engine defined by the LLMEngine protocol. The production implementation, TritonLLMEngine, checks readiness (line 179), maps OpenAI requests onto Triton tensors, and consumes the async gRPC stream we saw above, converting backend outputs into OpenAI schema via the response mapper. Because the engine is a protocol, a deployment can point the frontend at any engine that speaks it — a clean seam between chat semantics and inference mechanics.

What the Core Split Changes

It is worth pausing on the repository split, because it changes how you read everything else. Older write-ups describe src/core/ in this repository — the scheduler, the model lifecycle manager, the dynamic batcher. None of that is here anymore: TRITONSERVER_ServerNew links against a prebuilt libtritonserver, and the code you are left with is the part users actually touch. The effect on the source is visible immediately. The frontends cannot peek into engine internals, so every capability they expose — health, stats, shared memory, streaming — exists as a named C API function, which keeps the protocol implementations honest about what they promise.

The cost is coordination. Flags like the cache options parsed by ParseCacheConfigOption must be understood by both repositories, release branches must advance in lockstep, and an operator upgrading one without the other gets exactly the kind of version-skew failure the SIGNAL handler in main.cc cannot catch. It is the same trade every ecosystem eventually faces — monolith with invisible boundaries versus split with explicit contracts — and Triton chose the contract. Reading this repository after that split is therefore a study in what a stable extension surface looks like: the C API boundary in main.cc, the model repository layout on disk, and the protocol paths we walked above.

From Install to Your First Request

The practical path is Docker-first: nvcr.io/nvidia/tritonserver:<ver>-py3 ships the server, and a model repository — one directory per model with a config.pbtxt — is all it needs. Start the container, and the same server immediately serves HTTP on 8000, gRPC on 8001, and metrics on 8002. Client-side, registering a CUDA shared memory region before a big image classification burst measurably cuts per-request latency versus body-based transfer; the manager API we toured is the server half of that handshake. For LLM-shaped loads, point the OpenAI frontend at a tensorrtllm or vLLM backend model and you get a compliant chat-completions endpoint whose batching remains the engine’s business, not the frontend’s.

Being honest about the limits, because the source is: the core split means server on its own does not build — you fetch libtritonserver from the sibling repository, and version skews between the two are a real operational concern. The OpenAI frontend is young compared to dedicated LLM servers; its tool-call parsers and streaming behaviors are still evolving, and for pure LLM workloads TGI or vLLM integrate their engines more directly. Triton’s sweet spot remains heterogeneous fleets — vision, speech, embeddings, and LLMs behind one control plane.

That control-plane role is the takeaway. llama.cpp embeds into anything but serves one model class; TVM compiles kernels but has no server; Triton refuses to do either job and instead perfects the choreography between them. For the index of every serving stack we have toured, see the inference overview.

Enjoyed this post? Never miss out on future posts by following us