A Python Compiler for the GPU - Inside openai/triton
Writing a fast GPU kernel by hand means warp shuffles, shared memory swizzling, and a week in the profiler, which is why most deep learning teams would rather not. Triton, started by Philippe Tillet and described in the 2019 MAPL...
FP8 Without Rewriting Your Stack - Inside NVIDIA/TransformerEngine
Training large models in FP8 promises roughly double the throughput and half the memory traffic of BF16, and the 2022 paper FP8 Formats for Deep Learning is the reason the industry believes it. The catch has always been numerics: eight-bit...
Post-Training, the PyTorch Way - Inside pytorch/torchtune
Fine-tuning an LLM used to mean choosing between a framework that hid PyTorch from you and a notebook that hid correctness from you. Torchtune, released by Meta’s PyTorch team in April 2024 under the BSD 3-Clause license, took a third...
One Framework, Four Kinds of Retrieval - Inside UKPLab/sentence-transformers
In 2019, two researchers at the UKP Lab in TU Darmstadt published Sentence-BERT and open-sourced the code that made transformer embeddings practical for sentences. Six years later that codebase, UKPLab/sentence-transformers under the Apache-2.0 license and now maintained within Hugging Face,...
Your Science, Their Engineering - Inside Lightning-AI/pytorch-lightning
Every PyTorch project eventually grows the same skeleton: a training function that moves batches to the device, zeroes gradients, calls backward, clips gradients, steps the optimizer, logs metrics, validates at intervals, saves checkpoints, and then gets rewritten when someone says...
Train 0.19 Percent of the Model - Inside huggingface/peft
Full fine-tuning is the luxury the rest of us cannot afford. A 12B-parameter model in fp16 needs around 47GB of GPU memory before the optimizer even gets its share, which is why so many promising fine-tuning projects died the moment...
The Scan That Outgrew the Paper - Inside state-spaces/mamba
Every sequence model you have served pays attention’s tax: quadratic cost in sequence length, a KV cache that grows with every token, and memory bandwidth that becomes the bottleneck long before the FLOPs do. Mamba, from Albert Gu and Tri...
The Library That Indexes Billions - Inside facebookresearch/faiss
Every recommendation feed, duplicate detector, and retrieval-augmented answer starts with the same question: of these billion vectors, which few are closest to mine? Faiss, started by Herve Jegou at Meta’s Fundamental AI Research group and maintained by an author list...
One Booster, Every Parallelism - Inside hpcaitech/ColossalAI
Training a modern large model is a combinatorics problem disguised as a systems problem: data parallel or tensor parallel or stage parallel, optimizer states on GPU or CPU, activations recomputed or offloaded, and every combination trades memory against throughput differently....
Five Lines to Any Hardware - Inside huggingface/accelerate
Every PyTorch practitioner eventually hits the same wall: the training loop that ran beautifully on a laptop needs distributed ranks, gradient synchronization, loss scaling, and device juggling the moment it meets real hardware, and the boilerplate for all of that...
One Command to a Serving Cluster - Inside xorbitsai/inference
Most inference servers begin and end with “run this model on this GPU.” Xinference, from the team behind the Xorbits data-processing framework, starts from a different premise: model serving is a cluster problem, and it should feel as routine as...
A Deep Learning Stack That Fits in Your Head - Inside tinygrad/tinygrad
Somewhere between PyTorch’s twenty million lines and micrograd’s two hundred lies the sweet spot for understanding how deep learning frameworks actually work. tinygrad, maintained by tiny corp and released under the MIT license, plants its flag exactly there: an end-to-end...
The Smallest Load-Bearing Code in AI - Inside openai/tiktoken
Few repositories carry more traffic per line of code than openai/tiktoken. It is the fast byte pair encoding tokenizer behind OpenAI’s models — four lines of Python that virtually every cost estimate, context-window check, and prompt-length guard in the ecosystem...
The Server That Powered Hugging Chat - Inside huggingface/text-generation-inf...
Before every lab had its own serving stack, Hugging Face had to solve the problem for the whole ecosystem. Text Generation Inference, Apache-2.0-licensed and written in Rust with a Python GPU core, is the toolkit that powered Hugging Chat, the...
The App That Democratized AI Images - Inside AUTOMATIC1111/stable-diffusion-w...
Few open-source projects have shaped how ordinary people use AI models as much as AUTOMATIC1111’s stable-diffusion-webui. It turned Stable Diffusion from a Python notebook exercise into a one-click desktop application: clone, run a script, open a browser tab, and generate....
The Serving Engine Named After Its Cache - Inside sgl-project/sglang
Most inference engines are named after speed. SGLang is named after a data structure. The project from LMSYS grew out of RadixAttention, the idea that every prompt your users send is a prefix that has probably been seen before —...
The Machine-Written SDK - Inside openai/openai-python
Nearly every Python developer who has touched an AI product has typed the same two lines: import the openai package and instantiate a client. What few of them have considered is what those two lines actually invoke. The openai-python repository...
One Binary Between You and the Weights - Inside ollama/ollama
Ask a million developers what “running a model locally” looks like and most will describe one command: ollama run. The project by the same name is the software behind that moment, and its repository is a masterclass in hiding enormous...
The 300-Line GPT Classroom - Inside karpathy/nanoGPT
Most “minimal GPT” repositories are minimal the way a demo reel is minimal: the training script is a toy, and everything real is hidden in a framework you must first learn. karpathy/nanoGPT refuses that bargain. It is a rewrite of...
One Model, Every Device - Inside mlc-ai/mlc-llm
Most inference frameworks ask you to meet the model on the model’s terms: a datacenter GPU, a CUDA driver, a Python runtime, and a lot of VRAM. mlc-llm from the mlc-ai team flips that arrangement. It treats a large language...
The 13-File Transformer - Inside mistralai/mistral-inference
Production inference stacks have grown into sprawling systems, which makes Mistral AI’s own reference implementation almost contrarian. The mistral-inference repository — now archived, with the company pointing users at its API and vLLM-class servers — runs the full model family...
Two Engines, One Server - Inside InternLM/lmdeploy
LMDeploy from the InternLM team at Shanghai AI Laboratory occupies an interesting spot in the inference landscape. It ships two independent inference engines — a pure-PyTorch engine you can read and hack like any research codebase, and TurboMind, a C++...
A Neural Network With Nowhere to Hide - Inside karpathy/llm.c
Every deep learning framework makes the same bargain: you trade visibility for convenience, and after a few years you can ship models without ever seeing a training loop. llm.c, Andrej Karpathy’s follow-up to nanoGPT, tears up that bargain. It trains...
One Token That Unlocks the Image - Inside haotian-liu/LLaVA
Every multimodal assistant that chats about images owes something to this repository. LLaVA — Large Language and Vision Assistant — is the research codebase behind visual instruction tuning, the NeurIPS 2023 oral that showed a frozen CLIP encoder, a frozen...