Tokenizers Source Tour: The Rust Engine Behind Every HF Tokenizer
Every request that reaches a large language model is translated twice by the same unglamorous layer: from text into token ids on the way in, and from ids back to text on the way out. When you load a model...
Safetensors Source Tour: The Zero-Copy Weight Container
Every large model you download today is, on disk, mostly just a bag of numbers. The interesting engineering question is what wraps around that bag. For years the answer in the PyTorch world was torch.save, which wraps Python pickle —...
PyTorch Source Tour: One Op Call from Python to the Caching Allocator
Almost every framework we have toured on this blog sits on the same substrate. vLLM [(/vllm-2/)] spends its effort on KV-cache management and batch scheduling, but its tensor math is PyTorch. TorchServe [(/torchserve/)] and ExecuTorch [(/executorch/)] exist to package and...
Outlines Source Tour: Compiling JSON Schemas into Logit Masks
Ask a language model for JSON and it will happily hand you a stray brace, a trailing comma, or a confidently hallucinated field name. Structured generation attacks the problem at the root: instead of parsing and repairing whatever the model...
NCNN Source Tour: The Mobile-First C++ Inference Framework
Most inference frameworks we have toured on this blog — ONNX Runtime, OpenVINO, TVM — are desktop-first projects that were later coaxed onto phones. Tencent’s ncnn went the other way. It was born in 2017 for QQ and WeChat-style mobile...
MLX Source Tour: A NumPy-Shaped Array Library with a Unified Metal/CPU Backend
MLX is Apple’s array framework for machine learning on Apple silicon, and it occupies an interesting spot in the ecosystem. Next to PyTorch it looks deliberately small; next to JAX it shares the functional taste — function transformations like grad,...
JAX Source Tour: Tracing to Execution
Most numerical Python gives you two bad choices. Either you write eager code that is easy to debug but slow, or you write a static graph that is fast but alien. JAX’s answer is a third road: keep writing NumPy-style...
FasterTransformer Source Tour: The CUDA Kernel Lab Behind Modern LLM Serving
Between 2019 and 2022, if you wanted transformer inference to actually go fast on NVIDIA GPUs, you ended up reading FasterTransformer. It was NVIDIA’s C++/CUDA laboratory for the transformer block: attention, FFN, layernorm, sampling, and weight quantization treated as kernel-engineering...
After the Compiler Finishes - The Relax Virtual Machine Runtime of apache/tvm
Our first walk of apache/tvm followed the compiler’s happy path: Relax programs built by the BlockBuilder, rewritten by fifty-six C++ passes, lowered into TensorIR, and scheduled by dlight rules. That tour stopped, deliberately, at the boundary marked “executable” — the...
NVIDIA Triton Inference Server Source Tour: One Binary, Many Frontends
Most inference engines we have toured — vLLM, SGLang, TensorRT-LLM — are built around a single strong idea: batching. NVIDIA’s Triton Inference Server is built around a different one: being the front door. It speaks half a dozen protocols, runs...
From Python Call to cuLaunchKernelEx - The Triton Launch Path
Our first walk of openai/triton followed the compiler’s front door: how a decorated Python function becomes an AST, then TTIR, then the GPU-shaped dialects, then PTX or GCN. That leaves the other half of the story untouched — the part...
TorchServe Source Tour: A JVM Frontend Driving Python Workers Over a Socket P...
Most serving stacks we have toured live in one language: vLLM is Python down to its CUDA extensions, llama.cpp is C end to end. TorchServe — the official PyTorch model server, built with AWS — makes a bolder split: a...
The Quiet Workhorse of RAG - Inside huggingface/text-embeddings-inference
Every retrieval system has a generation model that gets the applause and an embedding model that does the work, and the embedding side is exactly where huggingface/text-embeddings-inference lives. TEI — as everyone calls it — is a Rust server whose...
Petals Source Tour: Serving One Giant Model Across a Swarm of Volunteer GPUs
Every serving stack we have toured owns its hardware. vLLM assumes a datacenter GPU, llama.cpp assumes your laptop, KTransformers assumes one machine with a big CPU and a modest GPU. Petals throws away that assumption entirely: it runs a single...
ONNX Runtime GenAI Source Tour: One C++ Core Wearing Two Hats
Most repositories we have toured on this site do one job well. ONNX defines a model format. llama.cpp runs quantized matrices on a laptop. vLLM serves fleets of requests. onnxruntime-genai, Microsoft’s generative-AI layer over ONNX Runtime, is unusual because it...
From Run() to Buffers - The ONNX Runtime Execution Machinery
Our first walk of microsoft/onnxruntime stayed at the architectural level: the versioned C API, the fusion transformers, MLAS, the CUDA provider, and the training stack. That post ends where every session ends up on a hot path — a call...
ONNX Source Tour: The Four Protobuf Messages Underneath Every Inference Stack
Read enough of our source tours and a pattern emerges: MIGraphX starts by parsing an ONNX file, IREE compiles from ONNX inputs, Triton Inference Server loads ONNX backends, and ONNX Runtime is literally named after it. The onnx repository itself...
Engine Actions and the Token Treadmill - The C++ Serve Core of mlc-ai/mlc-llm
Our first walk of mlc-ai/mlc-llm told the compile-and-serve story at the repository scale: passes that fuse weight dequantization into matmuls, quantization schemes, the async Python engine, and a note that a compact C++ core does the real work. That C++...
MIGraphX Source Tour: The AMD Graph Compiler That Turns ONNX Into HIP Kernels
NVIDIA users get a dizzying menu of inference stacks — we have toured TensorRT-LLM, vLLM, and FlashInfer on that side of the fence. On AMD’s ROCm platform the answer to “how does an ONNX model become fast” is more concentrated:...
Inside TurboMind: The C++ Engine Core of InternLM/lmdeploy
Our first visit to InternLM/lmdeploy stayed above the waterline: the CLI, the lite quantization toolkit, the OpenAI-compatible server, and the fork where a request picks either the readable PyTorch engine or the compiled TurboMind core. This visit goes under the...
KTransformers Source Tour: How kt-kernel Runs Giant MoE Models on CPU Plus GPU
KTransformers earned its reputation by doing something that sounds impossible: running DeepSeek-class mixture-of-experts models on a workstation GPU paired with a big pile of ordinary DDR memory. The trick is placement, not magic — dense, attention-critical work lives on the...
IREE Source Tour: One MLIR Compiler Stack From PyTorch Graph to Every Device
Most inference projects we have toured pick one strategy and sharpen it: vLLM optimizes GPU serving, llama.cpp squeezes CPUs, KTransformers splits work across both. IREE (Intermediate Representation Execution Environment) takes the hardest and most general path: a full compiler stack,...
GGML Source Tour: The Tiny C Engine That Runs llama.cpp
When we toured llama.cpp, the model layer got the attention — templated attention, KV-cache tricks, speculative decoding. Underneath all of it sits a separate repository that does the actual arithmetic: ggml, the tensor engine written in C that llama.cpp consumes...
FlashInfer Source Tour: The Plan/Run Machine Behind Fast Attention
Every time you chat with a model served by vLLM or SGLang, there is a good chance the attention step was executed by FlashInfer. The library sits one layer below the engines: engines decide what to run, FlashInfer decides how...