After the Compiler Finishes - The Relax Virtual Machine Runtime of apache/tvm
Our first walk of apache/tvm followed the compiler’s happy path: Relax programs built by the BlockBuilder, rewritten by fifty-six C++ passes, lowered into TensorIR, and scheduled by dlight rules. That tour stopped, deliberately, at the boundary marked “executable” — the...
NVIDIA Triton Inference Server Source Tour: One Binary, Many Frontends
Most inference engines we have toured — vLLM, SGLang, TensorRT-LLM — are built around a single strong idea: batching. NVIDIA’s Triton Inference Server is built around a different one: being the front door. It speaks half a dozen protocols, runs...
From Python Call to cuLaunchKernelEx - The Triton Launch Path
Our first walk of openai/triton followed the compiler’s front door: how a decorated Python function becomes an AST, then TTIR, then the GPU-shaped dialects, then PTX or GCN. That leaves the other half of the story untouched — the part...
TorchServe Source Tour: A JVM Frontend Driving Python Workers Over a Socket P...
Most serving stacks we have toured live in one language: vLLM is Python down to its CUDA extensions, llama.cpp is C end to end. TorchServe — the official PyTorch model server, built with AWS — makes a bolder split: a...
The Quiet Workhorse of RAG - Inside huggingface/text-embeddings-inference
Every retrieval system has a generation model that gets the applause and an embedding model that does the work, and the embedding side is exactly where huggingface/text-embeddings-inference lives. TEI — as everyone calls it — is a Rust server whose...
Petals Source Tour: Serving One Giant Model Across a Swarm of Volunteer GPUs
Every serving stack we have toured owns its hardware. vLLM assumes a datacenter GPU, llama.cpp assumes your laptop, KTransformers assumes one machine with a big CPU and a modest GPU. Petals throws away that assumption entirely: it runs a single...
ONNX Runtime GenAI Source Tour: One C++ Core Wearing Two Hats
Most repositories we have toured on this site do one job well. ONNX defines a model format. llama.cpp runs quantized matrices on a laptop. vLLM serves fleets of requests. onnxruntime-genai, Microsoft’s generative-AI layer over ONNX Runtime, is unusual because it...
From Run() to Buffers - The ONNX Runtime Execution Machinery
Our first walk of microsoft/onnxruntime stayed at the architectural level: the versioned C API, the fusion transformers, MLAS, the CUDA provider, and the training stack. That post ends where every session ends up on a hot path — a call...
ONNX Source Tour: The Four Protobuf Messages Underneath Every Inference Stack
Read enough of our source tours and a pattern emerges: MIGraphX starts by parsing an ONNX file, IREE compiles from ONNX inputs, Triton Inference Server loads ONNX backends, and ONNX Runtime is literally named after it. The onnx repository itself...
Engine Actions and the Token Treadmill - The C++ Serve Core of mlc-ai/mlc-llm
Our first walk of mlc-ai/mlc-llm told the compile-and-serve story at the repository scale: passes that fuse weight dequantization into matmuls, quantization schemes, the async Python engine, and a note that a compact C++ core does the real work. That C++...
MIGraphX Source Tour: The AMD Graph Compiler That Turns ONNX Into HIP Kernels
NVIDIA users get a dizzying menu of inference stacks — we have toured TensorRT-LLM, vLLM, and FlashInfer on that side of the fence. On AMD’s ROCm platform the answer to “how does an ONNX model become fast” is more concentrated:...
Inside TurboMind: The C++ Engine Core of InternLM/lmdeploy
Our first visit to InternLM/lmdeploy stayed above the waterline: the CLI, the lite quantization toolkit, the OpenAI-compatible server, and the fork where a request picks either the readable PyTorch engine or the compiled TurboMind core. This visit goes under the...
KTransformers Source Tour: How kt-kernel Runs Giant MoE Models on CPU Plus GPU
KTransformers earned its reputation by doing something that sounds impossible: running DeepSeek-class mixture-of-experts models on a workstation GPU paired with a big pile of ordinary DDR memory. The trick is placement, not magic — dense, attention-critical work lives on the...
IREE Source Tour: One MLIR Compiler Stack From PyTorch Graph to Every Device
Most inference projects we have toured pick one strategy and sharpen it: vLLM optimizes GPU serving, llama.cpp squeezes CPUs, KTransformers splits work across both. IREE (Intermediate Representation Execution Environment) takes the hardest and most general path: a full compiler stack,...
GGML Source Tour: The Tiny C Engine That Runs llama.cpp
When we toured llama.cpp, the model layer got the attention — templated attention, KV-cache tricks, speculative decoding. Underneath all of it sits a separate repository that does the actual arithmetic: ggml, the tensor engine written in C that llama.cpp consumes...
FlashInfer Source Tour: The Plan/Run Machine Behind Fast Attention
Every time you chat with a model served by vLLM or SGLang, there is a good chance the attention step was executed by FlashInfer. The library sits one layer below the engines: engines decide what to run, FlashInfer decides how...
Inside the Seek Loop - Timestamps, Fallback, and Hallucination Guards in SYST...
Our first walk of SYSTRAN/faster-whisper introduced the cast: WhisperModel, the batched wrapper, PyAV audio, the Silero VAD, and the claim that a reimplementation on CTranslate2 makes the same model four times faster. What that tour did not open up is...
From Export to Bare Metal - The .pte Path Through pytorch/executorch
Every framework we have toured in this series — Triton for kernels, vLLM for throughput, MLC for cross-platform compilation — assumes you have an operating system, a Python runtime, and memory to spare. ExecuTorch, PyTorch’s edge sibling, drops all three...
Inference Rebuilt for Batching - The Ragged Engine of DeepSpeed Inference v2
Our first walk of microsoft/DeepSpeed stayed on the training side of the tree: ZeRO’s three sharding stages, the pipe engine’s micro-batch schedules, the launcher that SSHes into clusters, and the accelerator layer that speaks for six hardware vendors. That tour...
Inside Microsoft's BitNet: Ternary Lookup-Table Kernels on CPU and CUDA
Two years of quantization work have convinced most teams that 4-bit is the practical floor for running large language models on everyday hardware. Microsoft’s BitNet refuses that floor entirely. Its weights live in {-1, 0, +1}, roughly 1.58 bits each,...
vLLM: Speculative Drafts and Grammar Gates - A Second Tour of vllm-project/vllm
Our first tour of vLLM walked the main road: requests enter the OpenAI-compatible server, the scheduler batches them, and the executor drives GPU workers that run the model. That post left two guard towers unvisited, and they turn out to...
Monkey-Patch Your Way to 2x Training - Inside unslothai/unsloth
Most fine-tuning libraries you meet are wrappers: they take Hugging Face Trainer, add a config file, and hope the stock kernels are fast enough. Unsloth is the opposite bet. Instead of orchestrating around transformers, it reaches inside them, replacing attention,...
The Scheduler Moved to Rust: A Second Tour of huggingface/text-generation-inf...
Our first tour of TGI at /tgi/ told the maintenance-mode story: a server that officially stepped back and let the ecosystem bloom around it. That framing, while accurate on the marketing page, undersells the code, because the main branch is...
TensorRT-LLM: Many Processes, One Engine - Inside the Executor Lattice
Our first tour of TensorRT-LLM followed the classic question: how does a request become tokens? We walked the configuration contract in LlmArgs, the two engine paths, the C++ batch manager with its KV cache arithmetic, and the serving tier on...