XNNPACK Source Tour: The Microkernel Library Powering Mobile Inference
If you have ever run a neural network on a phone, chances are the actual arithmetic was done by XNNPACK. It is the C library that sits underneath TensorFlow Lite, ExecuTorch, and the mobile builds of other runtimes, taking care...
torchvision Under the Hood: transforms v2 and the C++ Ops That Power Detection
Most people treat torchvision as a convenience layer: a Resize here, a pretrained ResNet there. But if you train or deploy detection models, you are standing on a much more interesting machine. The torchvision.transforms.v2 package rebuilt augmentation around a single...
torchaudio Source Tour: TorchCodec I/O, Custom Autograd, and the C++/CUDA Ker...
Most people meet torchaudio as a one-liner: torchaudio.load("clip.wav") returns a tensor and a sample rate, and that is the end of the story. Under the hood, though, the pytorch/audio repository is a fascinating case study in how a PyTorch domain...
Inside TensorFlow 2: Eager Execution, tf.function Tracing, and the Grappler M...
TensorFlow has a strange reputation problem. Ten years ago it was the framework everyone complained about: graphs, sessions, feed dictionaries, a learning curve shaped like a wall. TensorFlow 2 deleted almost all of that. There is no tf.Session anymore, no...
Optimum: How HuggingFace Routes Transformers Models to Every Accelerator
If you have ever typed pip install optimum[onnxruntime] and watched a PyTorch checkpoint turn into an ONNX Runtime model with one class swap, you have met Optimum at its best. Optimum is HuggingFace’s optimization toolbox that sits beside Transformers as...
open_clip: A Source Tour of the Maintained Open CLIP Implementation
When OpenAI released CLIP in 2021 it changed how the community thinks about vision-language pretraining: two towers, a temperature-scaled contrastive objective, and zero-shot classification from prompt templates. The original repository — which we walked through in the earlier tour of...
NVIDIA NeMo: A Source Tour of the Typed-Module Architecture Behind Speech and...
NVIDIA’s NeMo (Neural Modules) is one of those frameworks that quietly backs a large fraction of the open speech models you have heard of — Parakeet, FastConformer, Canary, Magpie — while also carrying NVIDIA’s newer speech-aware language models. Unlike a...
MaxText: Inside Google's JAX Blueprint for TPU-Scale LLM Training and Serving
Most LLM frameworks you meet on this blog start from PyTorch and bend toward accelerators. MaxText starts from the opposite shore: it is a pure Python / JAX reference implementation built first for Google Cloud TPUs and GPUs, and it...
gemma.cpp: Inside Google's Tiny C++ Inference Engine for Gemma Models
Most production LLM serving stacks are enormous: graph compilers, paged KV caches, continuous batching, speculative decoding. Google’s gemma.cpp walks in from the opposite direction. It is a lightweight, standalone C++ inference engine built for exactly one job — running the...
OpenAI CLIP Source Tour: Inside the Dual Encoder That Learned to Match Images...
When OpenAI released CLIP (Contrastive Language-Image Pre-training) in January 2021, the accompanying paper made a claim that sounded almost careless: train on 400 million (image, text) pairs scraped from the web, and the resulting model can name things it was...
TT-Metal Source Tour: Two Stacks on One RISC-V Accelerator
Most inference frameworks we have toured on this blog, like vLLM for serving, llama.cpp for local chat, or PyTorch itself as the universal substrate, sit on top of somebody else’s hardware abstraction. Tenstorrent’s tt-metal is the opposite kind of project:...
Tokenizers Source Tour: The Rust Engine Behind Every HF Tokenizer
Every request that reaches a large language model is translated twice by the same unglamorous layer: from text into token ids on the way in, and from ids back to text on the way out. When you load a model...
Safetensors Source Tour: The Zero-Copy Weight Container
Every large model you download today is, on disk, mostly just a bag of numbers. The interesting engineering question is what wraps around that bag. For years the answer in the PyTorch world was torch.save, which wraps Python pickle —...
PyTorch Source Tour: One Op Call from Python to the Caching Allocator
Almost every framework we have toured on this blog sits on the same substrate. vLLM [(/vllm-2/)] spends its effort on KV-cache management and batch scheduling, but its tensor math is PyTorch. TorchServe [(/torchserve/)] and ExecuTorch [(/executorch/)] exist to package and...
Outlines Source Tour: Compiling JSON Schemas into Logit Masks
Ask a language model for JSON and it will happily hand you a stray brace, a trailing comma, or a confidently hallucinated field name. Structured generation attacks the problem at the root: instead of parsing and repairing whatever the model...
NCNN Source Tour: The Mobile-First C++ Inference Framework
Most inference frameworks we have toured on this blog — ONNX Runtime, OpenVINO, TVM — are desktop-first projects that were later coaxed onto phones. Tencent’s ncnn went the other way. It was born in 2017 for QQ and WeChat-style mobile...
MLX Source Tour: A NumPy-Shaped Array Library with a Unified Metal/CPU Backend
MLX is Apple’s array framework for machine learning on Apple silicon, and it occupies an interesting spot in the ecosystem. Next to PyTorch it looks deliberately small; next to JAX it shares the functional taste — function transformations like grad,...
Keras Source Tour: One API, Three Backends
Keras spent a decade as TensorFlow’s friendly face. Keras 3 tore that coupling out: the same keras.Sequential, the same model.fit(), the same .keras file now run on TensorFlow, JAX, or PyTorch, selected with a single environment variable. That promise is...
JAX Source Tour: Tracing to Execution
Most numerical Python gives you two bad choices. Either you write eager code that is easy to debug but slow, or you write a static graph that is fast but alien. JAX’s answer is a third road: keep writing NumPy-style...
FasterTransformer Source Tour: The CUDA Kernel Lab Behind Modern LLM Serving
Between 2019 and 2022, if you wanted transformer inference to actually go fast on NVIDIA GPUs, you ended up reading FasterTransformer. It was NVIDIA’s C++/CUDA laboratory for the transformer block: attention, FFN, layernorm, sampling, and weight quantization treated as kernel-engineering...
After the Compiler Finishes - The Relax Virtual Machine Runtime of apache/tvm
Our first walk of apache/tvm followed the compiler’s happy path: Relax programs built by the BlockBuilder, rewritten by fifty-six C++ passes, lowered into TensorIR, and scheduled by dlight rules. That tour stopped, deliberately, at the boundary marked “executable” — the...
NVIDIA Triton Inference Server Source Tour: One Binary, Many Frontends
Most inference engines we have toured — vLLM, SGLang, TensorRT-LLM — are built around a single strong idea: batching. NVIDIA’s Triton Inference Server is built around a different one: being the front door. It speaks half a dozen protocols, runs...
From Python Call to cuLaunchKernelEx - The Triton Launch Path
Our first walk of openai/triton followed the compiler’s front door: how a decorated Python function becomes an AST, then TTIR, then the GPU-shaped dialects, then PTX or GCN. That leaves the other half of the story untouched — the part...
TorchServe Source Tour: A JVM Frontend Driving Python Workers Over a Socket P...
Most serving stacks we have toured live in one language: vLLM is Python down to its CUDA extensions, llama.cpp is C end to end. TorchServe — the official PyTorch model server, built with AWS — makes a bolder split: a...