The Machine Learning Compiler That Speaks Python - Inside apache/tvm
Most deep learning systems ship as fixed inference engines: you bring a model, the engine runs it, and the optimizations are whatever the vendor froze into the release. Apache TVM is the opposite bet. Its README calls it an Open...
A Python Compiler for the GPU - Inside openai/triton
Writing a fast GPU kernel by hand means warp shuffles, shared memory swizzling, and a week in the profiler, which is why most deep learning teams would rather not. Triton, started by Philippe Tillet and described in the 2019 MAPL...
FP8 Without Rewriting Your Stack - Inside NVIDIA/TransformerEngine
Training large models in FP8 promises roughly double the throughput and half the memory traffic of BF16, and the 2022 paper FP8 Formats for Deep Learning is the reason the industry believes it. The catch has always been numerics: eight-bit...
Post-Training, the PyTorch Way - Inside pytorch/torchtune
Fine-tuning an LLM used to mean choosing between a framework that hid PyTorch from you and a notebook that hid correctness from you. Torchtune, released by Meta’s PyTorch team in April 2024 under the BSD 3-Clause license, took a third...
The Clean-Room Trainer That Scales Llama to 512 GPUs - Inside pytorch/torchtitan
Every serious lab eventually writes the same three files: a model, a sharding plan, and a training loop that quietly accretes hundreds of special cases for every parallelism combination. The PyTorch team looked at that pattern and decided to publish...
One Framework, Four Kinds of Retrieval - Inside UKPLab/sentence-transformers
In 2019, two researchers at the UKP Lab in TU Darmstadt published Sentence-BERT and open-sourced the code that made transformer embeddings practical for sentences. Six years later that codebase, UKPLab/sentence-transformers under the Apache-2.0 license and now maintained within Hugging Face,...
Your Science, Their Engineering - Inside Lightning-AI/pytorch-lightning
Every PyTorch project eventually grows the same skeleton: a training function that moves batches to the device, zeroes gradients, calls backward, clips gradients, steps the optimizer, logs metrics, validates at intervals, saves checkpoints, and then gets rewritten when someone says...
Train 0.19 Percent of the Model - Inside huggingface/peft
Full fine-tuning is the luxury the rest of us cannot afford. A 12B-parameter model in fp16 needs around 47GB of GPU memory before the optimizer even gets its share, which is why so many promising fine-tuning projects died the moment...
One API, Twenty-Five Accelerators - Inside microsoft/onnxruntime
Somewhere behind your bank’s fraud model, your phone’s photo search, a Windows Copilot feature, and half the vector databases on the market, the same C library is running the inference: ONNX Runtime, Microsoft’s cross-platform inference and training accelerator, MIT-licensed, at...
The Scan That Outgrew the Paper - Inside state-spaces/mamba
Every sequence model you have served pays attention’s tax: quadratic cost in sequence length, a KV cache that grows with every token, and memory bandwidth that becomes the bottleneck long before the FLOPs do. Mamba, from Albert Gu and Tri...
The Compressor That Feeds vLLM - Inside vllm-project/llm-compressor
Every quantized model you load into vLLM was made by something, and for most of the official checkpoints that something is llmcompressor - the vLLM project’s own optimization library. It takes a Hugging Face checkpoint plus a recipe, runs a...
20 LLMs, One Readable Codebase - Inside Lightning-AI/litgpt
Most LLM libraries hand you a wrapper around someone else’s model class and call it a day. LitGPT took the opposite bet: reimplement every popular open model from scratch - Llama 3, Gemma 2/3, Phi 4, Qwen2.5 and Qwen3, Mistral,...
The Kernel That Rewrote Attention - Inside Dao-AILab/flash-attention
Every transformer trained since 2022 owes part of its existence to a paper proof that attention could be computed without ever writing the full sequence-by-sequence matrix to HBM. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness by Tri Dao, Daniel...
The Library That Indexes Billions - Inside facebookresearch/faiss
Every recommendation feed, duplicate detector, and retrieval-augmented answer starts with the same question: of these billion vectors, which few are closest to mine? Faiss, started by Herve Jegou at Meta’s Fundamental AI Research group and maintained by an author list...
The Vector Search Engine Under the Databases - Inside rapidsai/cuvs
Every retrieval-augmented generation stack, every recommendation feed, and every semantic search box eventually asks the same question: given this vector, which of the hundred million stored vectors are its nearest neighbors? Apache-2.0 licensed cuVS, version 26.12.00 on the RAPIDS release...
The Template Cathedral of GEMM - Inside NVIDIA/cutlass
Every training run and every inference server on NVIDIA hardware eventually bottoms out in a matrix multiply, and for most of the industry that multiply comes from CUTLASS. Started in 2017 and cited by Vijay Thakkar and colleagues, the project...
The Runtime That Made Serving Transformers Cheap - Inside OpenNMT/CTranslate2
When translation or speech-recognition servers needed to handle real traffic on real hardware, the OpenNMT ecosystem discovered that general deep learning frameworks were the wrong tool: too much memory, too much overhead, too little control over precision. The answer is...
One Booster, Every Parallelism - Inside hpcaitech/ColossalAI
Training a modern large model is a combinatorics problem disguised as a systems problem: data parallel or tensor parallel or stage parallel, optimizer states on GPU or CPU, activations recomputed or offloaded, and every combination trades memory against throughput differently....
From Float32 to Four Bits - Inside pytorch/ao
Quantization libraries have a reputation for being bolt-on afterthoughts: a separate graph mode, a foreign config format, kernels that break the moment you wrap your model in torch.compile. TorchAO, launched by the PyTorch team in September 2024 and versioned 0.19.0...
Five Lines to Any Hardware - Inside huggingface/accelerate
Every PyTorch practitioner eventually hits the same wall: the training loop that ran beautifully on a laptop needs distributed ranks, gradient synchronization, loss scaling, and device juggling the moment it meets real hardware, and the boilerplate for all of that...
One Command to a Serving Cluster - Inside xorbitsai/inference
Most inference servers begin and end with “run this model on this GPU.” Xinference, from the team behind the Xorbits data-processing framework, starts from a different premise: model serving is a cluster problem, and it should feel as routine as...
A Deep Learning Stack That Fits in Your Head - Inside tinygrad/tinygrad
Somewhere between PyTorch’s twenty million lines and micrograd’s two hundred lies the sweet spot for understanding how deep learning frameworks actually work. tinygrad, maintained by tiny corp and released under the MIT license, plants its flag exactly there: an end-to-end...
The Smallest Load-Bearing Code in AI - Inside openai/tiktoken
Few repositories carry more traffic per line of code than openai/tiktoken. It is the fast byte pair encoding tokenizer behind OpenAI’s models — four lines of Python that virtually every cost estimate, context-window check, and prompt-length guard in the ecosystem...
The Server That Powered Hugging Chat - Inside huggingface/text-generation-inf...
Before every lab had its own serving stack, Hugging Face had to solve the problem for the whole ecosystem. Text Generation Inference, Apache-2.0-licensed and written in Rust with a Python GPU core, is the toolkit that powered Hugging Chat, the...