If you run GPU boxes for training, serve models behind inference gateways, or orchestrate batch jobs across a cluster, you already know the uncomfortable truth: things fail silently unless something is watching. A saturated NVLink, a queue that stops draining, a worker that leaks memory for six hours before the OOM killer arrives — none of these page you by themselves. That is exactly the gap a metrics system fills, and the de facto standard for filling it is Prometheus, the CNCF project that has become the connective tissue of cloud-native observability. When you pair it with dashboards or hook it into a Kubernetes deployment, every node, pod and GPU exporter in your fleet becomes a queryable time series.

What makes Prometheus interesting from a code perspective is that it is not a thin orchestrator gluing together other databases. It is a self-contained, single-binary system written in Go that ships its own time series database (TSDB), its own query language engine (PromQL), its own pull-based collection machinery, and its own rule evaluator for recording and alerting. Everything from the write-ahead log on disk to the AST evaluator that computes rate() over your metrics lives in one repository. For ML infrastructure teams, that matters: your training cluster metrics are stored in a format you can actually read the source code of, and the storage cost model is local disk, not a metered cloud service.

In this post we walk through the actual source of prometheus/prometheus on the main branch and map how the pieces fit together: the entry point in cmd/prometheus, the scrape manager, the TSDB write path, block compaction, the PromQL engine, rule evaluation, and the web API that ties it all together. Every function and line number below was verified directly against the checked-out source, and the two diagrams are generated from the real call structure. If you later want to give these metrics a SQL-flavored home for long-horizon analytics, our TimescaleDB tour is a natural companion read; for dashboards on top, see Grafana.

Overview: one Go binary wires the web API, scrape manager, rule manager and TSDB together at startup; scrapes feed the head block through the WAL, and compaction drains the head into immutable blocks that the query engine reads back.

Reading the overview from left to right:

• main() at cmd/prometheus/main.go, L389 parses flags — the config file defaults to prometheus.yml (L434) and the listen address to 0.0.0.0:9090 (L443) — then opens storage and starts every subsystem below it.
• scrape.NewManager at scrape/manager.go, L52 builds the collection side; its Run loop, L223 consumes target sets that service discovery produces and reconciles one scrape loop per target.
• scrapeLoop.run at scrape/scrape.go, L1404 is the per-target heartbeat: on every interval tick it calls scrapeAndReport, L1487, which fetches /metrics, decodes the payload, and even synthesizes the famous up metric when a target misbehaves.
• DB.Appender at tsdb/db.go, L1366 is the single write entrance; it wraps the head block’s transactional appender so that both scrape loops and rule evaluations funnel into the same path.
• WL.Log at tsdb/wlog/wlog.go, L657 writes each batch into the write-ahead log before the sample is considered durable — this is why a crash costs you seconds of data, not the whole head.
• Engine at promql/engine.go, L519 is the PromQL evaluator; exec, L1091 drives each query from AST down to storage reads.
• rules.Manager at rules/manager.go, L121 owns rule groups; each Group.Eval at rules/group.go, L504 runs on its own interval and pushes results straight back into storage.

Detailed view: the write path (scrape to WAL to head), the compaction path (head to leveled blocks), the read path (API to engine to queriers over head and blocks), and the rule loop that closes the circle.

From Install to Your First Query

The README keeps getting-started deliberately short, and we verified every claim here against it. The fastest route is the official Docker image: docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus, after which the README states Prometheus is reachable at http://localhost:9090/. If you prefer binaries, precompiled releases are on the prometheus.io download page, and building from source requires the Go version pinned in go.mod plus NodeJS for the web UI, then either go install github.com/prometheus/prometheus/cmd/... or make build followed by ./prometheus --config.file=your_config.yml.

Configuration is a single YAML file. The README links an example at documentation/examples/prometheus.yml, and the minimal shape most people start with is a global section setting scrape_interval and a scrape_configs list. Each job declares a static_configs block (or relies on one of dozens of service-discovery integrations); the agent side of that bargain is the Target type in scrape/target.go, L63, which resolves the final scrape URL, L234 and carries the label set that will identify the series forever.

For a first query, open the UI and evaluate up — you should see one series per target with a value of 1. From there the natural next steps are rate(container_cpu_usage_seconds_total[5m]) for CPU burn, or max_over_time(node_memory_MemAvailable_bytes[1h]) when a training host looks sluggish. Each of those goes through the exact engine path we dissect next. And if your fleet is containerized, the same scrape model extends cleanly to a Kubernetes cluster via pod annotations or its discovery mechanisms.

The Write Path: WAL First, Memory Second

When the process starts, Open at tsdb/db.go, L919 opens the data directory, loads persisted blocks, and rebuilds the head by replaying the log — Head.loadWAL at tsdb/head_wal.go, L82 reconstructs series and samples from WAL records so startup survives restarts. The head itself is built by NewHead at tsdb/head.go, L308 and exposed as the Head struct, L72: an in-memory map of series with memory-mapped chunks for older stretches.

Writes arrive as small transactions. Head.Appender at tsdb/head_append.go, L173 hands out an appender, and each call to headAppender.Append, L444 performs series lookup-or-create, ordering checks, and chunk encoding. The WAL wrapper (WL struct at tsdb/wlog/wlog.go, L182) is created with NewSize, L300, which fixes the segment size; WL.Log, L657 then pages, compresses and flushes records. The design point to remember: durability is cheap here because a WAL record is bytes, not an index update — the expensive indexing happens in memory and is rebuilt on replay.

One detail that shows how battle-tested this code is: the head keeps a bounded out-of-order window so late-arriving samples from a slow or retried scrape can still land correctly, a necessity when your scrape targets include network hops across a training cluster.

Compaction: From Head to Immutable Blocks

The head cannot grow forever. On a background cycle DB.Compact at tsdb/db.go, L1560 cuts the head into a block and then evaluates retention deletions. The strategy object behind this is the Compactor interface at tsdb/compact.go, L54, whose default implementation is the LeveledCompactor struct, L80, constructed through NewLeveledCompactor, L201. It merges blocks into progressively larger time ranges — by default the first level is two hours (DefaultBlockDuration at tsdb/db.go, L57), and the default retention configured in DefaultOptions at tsdb/db.go, L83 is 15 days (L87).

Blocks are immutable once written, which is what makes reads and backups tractable: a block is a directory with an index, chunk files and metadata, and queries reach it through the BlockReader interface at tsdb/block.go, L146. This layering — mutable head, immutable blocks, log in between — is the same conceptual shape you find in Redis persistence or log-structured stores, and it is why Prometheus can absorb tens of thousands of samples per second on commodity hardware without tuning.

How PromQL Evaluation Actually Works

The query side starts at Engine at promql/engine.go, L519, constructed by NewEngine, L568 with EngineOpts that bound sample counts and per-query timeout. The web API builds queries through two entry points: NewInstantQuery, L788 for a single timestamp and NewRangeQuery, L835 for start/end/interval evaluation. Both funnel into exec, L1091, which sets up query bookkeeping, and then into execEvalStmt, L1257, which opens storage queriers and walks the parsed AST with an evaluator.

The practical consequence: every vector selector in your expression becomes a series fetch against both the head (for recent samples) and persisted blocks (for history). Queries that scan wide label spaces pay for it in memory, which is why the MaxSamples guard exists. On the serving side, requests arrive at the v1 API (API struct at web/api/v1/api.go, L241): the instant handler query, L625 and the range handler queryRange, L1201 are registered on the router at L480-481, and the whole surface is mounted by the Handler in web/web.go, L222, whose serving loop lives in Run, L748.

Rules: Recording and Alerting

Rule files let you precompute expressions and define alerts. rules.NewManager at rules/manager.go, L172 builds the manager (Manager struct, L121); config reloads call Update, L271, which swaps in the new group set atomically. Each Group at rules/group.go, L45 runs its own goroutine (L208) on its configured interval, and Group.Eval, L504 evaluates rules in order so recording rules can feed alerts downstream.

Two rule kinds exist. RecordingRule at rules/recording.go, L39 evaluates an expression and writes the result as a new series (Eval, L86) — this is how you precompute expensive aggregations for dashboards. AlertingRule at rules/alerting.go, L116 runs the same evaluation but maintains a pending/firing state machine with for-clause tracking, and those states are exported as ALERTS series. For ML ops specifically, recording rules are the pragmatic answer to “what was p95 GPU memory across the fleet over the last hour” without re-aggregating raw series on every dashboard load — and they pair naturally with a scheduled retraining flow like the one we describe in the Airflow tour.

Honest Limits

Prometheus is deliberately not a distributed database. It is a single-node store by design: there is no built-in replication or sharding, and the project itself points users who need horizontal scale or long-term storage toward external systems (Thanos, Cortex, Mimir) that sit on top of it. If your retention ambition is measured in years, plan for object-storage-backed federation early rather than after your local disk fills.

Second, the pull model presumes scrapeable targets. Short-lived batch jobs — a five-minute training epoch, a one-off evaluation run — can finish before any scrape happens; the standard mitigation is a push gateway or converting job metrics into counters emitted by a longer-lived sidecar. Third, cardinality is the real operational cliff: every unique label combination is a series in memory and on disk, so an unbounded label like user_id or a full stack trace attached as a label will take the head down with it. Finally, PromQL is purpose-built for metrics math, not full-text logs or traces — it will not replace your log store, and trying to bend it into one is a common rookie mistake. None of these limits are bugs; they are the explicit trade-offs that keep the single binary fast, and the source we walked through shows exactly where each boundary is enforced.

{“@context”:”https://schema.org”,”@type”:”TechArticle”,”headline”:”Prometheus in Go: How the Monitoring Engine Behind Modern Infrastructure Actually Works”,”description”:”A source-level tour of the Prometheus TSDB, PromQL engine, scrape manager, rule evaluation and web API.”,”author”:{“@type”:”Person”,”name”:”PyShine”},”datePublished”:”2026-10-31”,”keywords”:”prometheus, monitoring, tsdb, promql, go”}

Enjoyed this post? Never miss out on future posts by following us