Inside runc: A Source Tour of the OCI Runtime That Boots Every AI Container
Every GPU inference workload you have ever run bottoms out in the same few thousand lines of Go. Whether a vLLM server is scheduled by Kubernetes, pulled by CRI-O, launched through containerd, or started by the Docker engine, the final...
Prometheus Operator Source Tour: How One Go Binary Turns CRDs Into StatefulSets
Run a fleet of LLM servers — vLLM, SGLang, Triton — and the first operational question after “does it answer?” is “what is it doing right now?”. GPU utilization, KV-cache pressure, request queue depth, tokens per second: all of that...
Inside Moby: A Source Tour of the Docker Engine Daemon
Every modern AI workload ships in a container. You pull a CUDA base image, layer on PyTorch or a quantized LLM runtime, and hand it to a scheduler — and the binary quietly doing the heavy lifting behind docker run...
Kustomize Source Tour: How Template-Free YAML Customization Actually Builds
Serving a large language model on Kubernetes rarely means writing one YAML file. A dev cluster runs one GPU replica with lenient probes; staging and production differ in replicas and limits — the resources are mostly identical while a handful...
Kubernetes From the Source: How kube-apiserver and kubelet Are Wired in Go
If you serve an LLM in production today, the odds are good that the model runs in a Pod somewhere on a Kubernetes cluster. The scheduler picked a GPU node, the node agent pulled the image, started the container, and...
Helm Source Tour: How the Kubernetes Package Manager Renders Charts and Store...
Serving an LLM today is rarely one container. A realistic inference stack means a model server such as vLLM or Triton, a GPU device plugin, autoscaling rules, service endpoints, and a pile of config that all have to land in...
Flux from the Source: How the GitOps Toolkit Bootstraps, Syncs and Heals a Cl...
Serving a large model in production is an operations problem as much as a modeling problem. A vLLM or Triton inference server is a Deployment, its weights are a PersistentVolume or an object store, its autoscaling is an HPA, and...
CRI-O Source Tour: How Kubernetes' Leanest Runtime Speaks CRI in Go
When you schedule a large language model pod onto a Kubernetes node, someone has to actually fork the container process, mount the model weights, wire the network namespace, and keep the thing supervised after the scheduler moves on. On many...
containerd Source Tour: The Daemon, the Shim Protocol, and the Snapshot Layer
Every AI inference service you deploy — a Triton server, a vLLM worker, or a plain FastAPI wrapper around a fine-tuned model — exists as a container on some Linux host. On most modern clusters, and on plenty of single-box...
Inside Argo CD's Source: How the GitOps Controller Turns Git Commits into Clu...
GitOps is the operating model where Git is the source of truth for what runs in your cluster, and a controller’s only job is to close the loop between the repository and the API server. For AI/ML infrastructure this matters...
Traefik in the Source: How a Go Reverse Proxy Learns Your Fleet and Rewires I...
If you run more than one model server — a vLLM box here, a FastAPI inference container there, a GPU node pool over in Kubernetes — then the most performance-critical piece of software you own is not the model server....
Linkerd: Reading the Rust and Go Source of the Lightest Service Mesh
Modern AI inference is quietly becoming a distributed-systems problem. A production stack today is rarely one container: there is a gateway, a tokenizer, a prefill-versus-decode split across GPU pods, a vector database, maybe a reranker, and a scrape target for...
Inside Kong: The Lua Gateway That Wraps Nginx into an AI Traffic Machine
If you have ever served a machine-learning model to real users, you have run into the same trio of problems: you need to throttle who can call your inference endpoint, authenticate those calls, and decide which model version a request...
Istio: Reading the Go Source of the Service Mesh Control Plane
Serving large language models in production is rarely one process talking to one GPU. A realistic AI platform is a swarm of Kubernetes workloads: a tokenizer service, one or more vLLM or Triton replicas, an embedding server, a vector database,...
ingress-nginx: Reading the Go Source of Kubernetes' Most-Deployed Ingress Con...
Almost every Kubernetes cluster that serves an AI workload leans on an ingress controller at its front door, and for years the default door has been ingress-nginx: the reference NGINX-based controller maintained under the Kubernetes umbrella. When you expose a...
HAProxy: Reading the C Source of the Internet's Favorite Load Balancer
Every serious AI serving stack leans on a load balancer somewhere in front of its GPU fleet. Whether you are spreading traffic across four vLLM replicas, fronting an Ollama box with sticky sessions, or just keeping a FastAPI model server...
Envoy: Reading the C++ Source of the Cloud-Native Edge Proxy
Ask anyone who runs LLM inference at scale who actually fronts their GPU fleet, and the answer is very often Envoy. The project — which describes itself on its README as a “cloud-native high-performance edge/middle/service proxy” — is the data...
Contour: Reading the Go Source of the Envoy-Powered Ingress Controller
Every GPU cluster faces the same front-door problem. Model servers such as vLLM, Triton, or SGLang expose HTTP and gRPC endpoints, traffic has to shift gradually from one model version to the next, and the request flow must stay fast...
Caddy: The Go Web Server Where HTTPS Is Not Your Job
Anyone who has put a machine learning model online knows the drill: you wrap a FastAPI app around your inference code, run it on port 8000, and then discover that the real work starts. Getting traffic to it over the...
Apache APISIX Under the Hood: The etcd-Driven Gateway Built for AI Traffic
Every serious AI stack hits the same wall eventually: model servers, vector databases, and agent backends all grow from a handful of services into a zoo of endpoints, each with its own auth scheme, its own rate limits, and its...
VictoriaMetrics: Reading the Go Source of a Time-Series Powerhouse
Anyone who has trained or served a model at scale knows the feeling: the GPU node is hot, the loss curve looks fine, and then the run silently stalls because some unattended metric — host memory, NVLink errors, an overloaded...
Valkey Source Tour: How the Community-Governed Redis Fork Rewired Its Core
In March 2024 the Redis project moved its source tree to a source-available license, and the open-source world responded the way open-source worlds do: it forked. The result is Valkey, a high-performance data structure server that, per its own README,...
Thanos: Giving Prometheus Unlimited Memory with Go and Object Storage
If you run even a modest AI or ML fleet, your metrics outgrow your monitoring faster than your models outgrow your GPUs. A single GPU export loop that scrapes utilization, memory pressure, KV-cache occupancy and request latency every fifteen seconds...
Inside the OpenTelemetry Collector: The Go Engine That Routes Every Trace, Me...
Modern AI systems are telemetry furnaces. An LLM gateway emits metrics that classic web apps never had: tokens per second, time-to-first-token, queue depth per model replica, cache hit ratios on a prompt store. Every step of an agent — tool...