Traefik in the Source: How a Go Reverse Proxy Learns Your Fleet and Rewires I...
If you run more than one model server — a vLLM box here, a FastAPI inference container there, a GPU node pool over in Kubernetes — then the most performance-critical piece of software you own is not the model server....
Linkerd: Reading the Rust and Go Source of the Lightest Service Mesh
Modern AI inference is quietly becoming a distributed-systems problem. A production stack today is rarely one container: there is a gateway, a tokenizer, a prefill-versus-decode split across GPU pods, a vector database, maybe a reranker, and a scrape target for...
Inside Kong: The Lua Gateway That Wraps Nginx into an AI Traffic Machine
If you have ever served a machine-learning model to real users, you have run into the same trio of problems: you need to throttle who can call your inference endpoint, authenticate those calls, and decide which model version a request...
Istio: Reading the Go Source of the Service Mesh Control Plane
Serving large language models in production is rarely one process talking to one GPU. A realistic AI platform is a swarm of Kubernetes workloads: a tokenizer service, one or more vLLM or Triton replicas, an embedding server, a vector database,...
ingress-nginx: Reading the Go Source of Kubernetes' Most-Deployed Ingress Con...
Almost every Kubernetes cluster that serves an AI workload leans on an ingress controller at its front door, and for years the default door has been ingress-nginx: the reference NGINX-based controller maintained under the Kubernetes umbrella. When you expose a...
HAProxy: Reading the C Source of the Internet's Favorite Load Balancer
Every serious AI serving stack leans on a load balancer somewhere in front of its GPU fleet. Whether you are spreading traffic across four vLLM replicas, fronting an Ollama box with sticky sessions, or just keeping a FastAPI model server...
Envoy: Reading the C++ Source of the Cloud-Native Edge Proxy
Ask anyone who runs LLM inference at scale who actually fronts their GPU fleet, and the answer is very often Envoy. The project — which describes itself on its README as a “cloud-native high-performance edge/middle/service proxy” — is the data...
Contour: Reading the Go Source of the Envoy-Powered Ingress Controller
Every GPU cluster faces the same front-door problem. Model servers such as vLLM, Triton, or SGLang expose HTTP and gRPC endpoints, traffic has to shift gradually from one model version to the next, and the request flow must stay fast...
Caddy: The Go Web Server Where HTTPS Is Not Your Job
Anyone who has put a machine learning model online knows the drill: you wrap a FastAPI app around your inference code, run it on port 8000, and then discover that the real work starts. Getting traffic to it over the...
Apache APISIX Under the Hood: The etcd-Driven Gateway Built for AI Traffic
Every serious AI stack hits the same wall eventually: model servers, vector databases, and agent backends all grow from a handful of services into a zoo of endpoints, each with its own auth scheme, its own rate limits, and its...
VictoriaMetrics: Reading the Go Source of a Time-Series Powerhouse
Anyone who has trained or served a model at scale knows the feeling: the GPU node is hot, the loss curve looks fine, and then the run silently stalls because some unattended metric — host memory, NVLink errors, an overloaded...
Valkey Source Tour: How the Community-Governed Redis Fork Rewired Its Core
In March 2024 the Redis project moved its source tree to a source-available license, and the open-source world responded the way open-source worlds do: it forked. The result is Valkey, a high-performance data structure server that, per its own README,...
Thanos: Giving Prometheus Unlimited Memory with Go and Object Storage
If you run even a modest AI or ML fleet, your metrics outgrow your monitoring faster than your models outgrow your GPUs. A single GPU export loop that scrapes utilization, memory pressure, KV-cache occupancy and request latency every fifteen seconds...
Inside the OpenTelemetry Collector: The Go Engine That Routes Every Trace, Me...
Modern AI systems are telemetry furnaces. An LLM gateway emits metrics that classic web apps never had: tokens per second, time-to-first-token, queue depth per model replica, cache hit ratios on a prompt store. Every step of an agent — tool...
NATS: The Featherweight Messaging Backbone for Distributed AI
Every serious AI system turns into a fan-out problem embarrassingly fast. One “answer the user” request becomes a burst of calls to an embedding service, a retriever, a reranker, and a model provider; a training run becomes millions of checkpoint,...
Inside Memcached: The C Event Machine That Still Wins the Latency Game
If you build anything around models, you spend your life answering the same question over and over: the same prompt prefix, the same tokenizer call, the same feature vector, the same session lookup. Doing that work twice is pure waste,...
Grafana Loki: Log Aggregation That Indexes Labels, Not Lines
If you have ever trained a model overnight and woken up to a cryptic CUDA OOM at 3 a.m., you already know the real product of an ML system is not just the checkpoint — it is the log trail....
Jaeger: Tracing Every Hop of Your Distributed AI Stack
Modern AI systems are distributed systems in disguise. A single “ask the model a question” request in a production RAG application usually touches an API gateway, an embedding service, a vector database, a retriever, maybe a reranker, and finally an...
InfluxDB 3: A Source Tour of the Rust, Arrow, and Parquet Time-Series Engine
Anyone who has trained a model or operated a GPU cluster knows the feeling: the metrics are everywhere and the answers are nowhere. GPU utilization counters, temperature probes, token latencies, loss curves from experiment trackers, queue depths from an inference...
etcd: The Tiny Consensus Machine Holding Kubernetes Together
If you train or serve machine-learning systems on Kubernetes, every meaningful thing you do passes through a small Go program you have probably never looked at. When the scheduler decides which node runs your GPU job, it reads and writes...
Typesense Source Tour: How the C++ Search Engine Is Wired, From HTTP to Inver...
Typesense is a compact, open-source search engine written in C++ that has quietly become the default answer for teams that need typo-tolerant, millisecond-latency search without operating a distributed JVM cluster. Its promise, repeated in the project’s own README, is simplicity:...
TimescaleDB Source Tour: How Hypertables, Chunk Routing, Columnstore, and Con...
TimescaleDB is a PostgreSQL extension for high-performance real-time analytics on time-series and event data, developed by Timescale (now Tiger Data). Unlike a fork of Postgres or a separate storage engine, it is a loadable extension written in C that hooks...
Tantivy source tour: how the Rust full-text search engine library works
Most search tutorials stop at the API surface. This one goes the other way: we open the crate itself. Tantivy is a fast full-text search engine library written in Rust — closer in spirit to Apache Lucene than to a...
SurrealDB: One Rust Binary for Documents, Graphs, and Vectors
If you have ever built an AI-flavored application, you know the drill: documents go into a document store, the relationship graph goes into a graph database, embeddings go into a dedicated vector engine such as Qdrant or Milvus, and then...