Every AI system eventually hits the same wall: your models are only as good, and only as fresh, as the events that reach them. Feature vectors generated a few seconds after a user click, GPU utilization samples scraped off a training node, inference requests and their latencies, token-by-token usage records from an LLM gateway — all of these arrive as continuous streams of facts, and batching them through cron jobs or message queues that silently drop data stops scaling fast. This is the gap Apache Kafka was built to fill, and it is why Kafka sits upstream of so many ML feature stores, real-time dashboards, and telemetry backends (often right next to a columnar sink like ClickHouse).

Kafka’s core idea is deceptively simple: a distributed, append-only commit log. Producers append records to the end of partitioned logs; consumers read them at their own pace by keeping an offset — a pointer into the log — rather than by deleting messages. Because the log is durable and replayable, the same stream can feed a live dashboard, a training-data dump into object storage, and an online feature extractor simultaneously, each with its own position. Fault tolerance comes from replicating each partition across brokers, with a high-watermark defining what has been safely copied and therefore what consumers are allowed to see.

In this post we read the actual implementation on the trunk branch of apache/kafka — the Scala/Java broker core plus its Java clients. Two architecture diagrams follow: a high-level overview, and a detailed map of the classes that carry every message from a producer’s send() call to a consumer’s poll(). All class and line references link into the real sources on GitHub, so you can follow along in your editor.

Overview: records flow from the producer's accumulator through the Sender to the broker's network layer, get dispatched by KafkaApis to the ReplicaManager, land in the UnifiedLog as indexed segments, while the group coordinator tracks consumers. Click any node to open the source on GitHub.

Reading the overview from left to right:

• KafkaProducer.send, L1027 serializes the record, runs it through interceptors, and picks a destination partition — either one you set, or computed in partition() at L1681.
• RecordAccumulator.drain, L1083 collects the batches that are ready to go, capped by the max request size per node.
• Sender.run, L241 is the producer’s I/O loop; it turns drained batches into ProduceRequest RPCs via sendProduceRequest, L896.
• SocketServer, L71 accepts connections with an Acceptor (L458) and reads and decodes requests in Processor threads (L779), enqueueing them into a bounded queue.
• KafkaRequestHandler.run, L108 pulls requests off that queue and dispatches them at L167 to the API layer.
• KafkaApis.handleProduceRequest, L401 is the protocol facade every client request crosses; fetches enter at handleFetchRequest, L568.
• ReplicaManager.appendRecordsToLeader, L586 checks the ISR and required acks, then routes the write through a Partition to the log.
• UnifiedLog.appendAsLeader, L1033 appends to the active segment and rolls it when it grows too large (maybeRoll, L2166).
• GroupCoordinatorService.joinGroup, L1009 runs consumer-group rebalancing and offset commits on the new KRaft-era coordinator runtime.

Detail: the producer's accumulator–Sender–NetworkClient chain, the consumer's delegate split (classic vs KIP-848 async), the broker's network/handler/replication tiers, the segment-and-index storage engine, and the coordinator's service–shard–metadata split.

The Broker: From Socket to Commit Log

The request path is deliberately boring, which is why it is fast. SocketServer (class at L71) multiplexes NIO sockets: one Acceptor per listener (L458) hands new connections to Processor threads (L779) that decrypt, parse frame headers, and enqueue complete requests. A fixed pool of KafkaRequestHandler worker threads — sized by num.io.threads and resizable at runtime through KafkaRequestHandlerPool, L227 — drains the queue held by RequestChannel, L40, and replies travel back through sendResponse, L91.

KafkaApis (class, L95) is the switchboard: produce requests go to handleProduceRequest (L401), fetches to handleFetchRequest (L568), offset reads to handleOffsetFetchRequest (L1029), and group membership to handleJoinGroupRequest (L1390). Each handler validates the request, calls into the right subsystem, and registers a callback that completes the response asynchronously — the handler thread is never blocked waiting for a disk write or a quorum ack.

The replication brain lives in ReplicaManager (class, L154). appendRecordsToLeader (L586) refuses writes if the partition has no elected leader or the ISR has shrunk below min.insync.replicas; the actual append funnels through appendToLocalLog (L1376) into per-partition state held by Partition (class, L163). Leadership transitions are explicit — makeLeader (L590) and makeFollower (L696) — triggered by metadata records from the KRaft controller. Reads flow through fetchMessages (L1665), which can park a fetch and defer the response until enough new data arrives, a trick that keeps consumer latency low without busy polling.

The Storage Engine: Segments, Indexes, Watermarks

One of the bigger architectural shifts of recent Kafka versions is that the log implementation moved from Scala into a plain-Java storage module. The Scala kafka.log.LogManager you find in core is now just a factory (object, L34) that instantiates the Java org.apache.kafka.storage.internals.log.LogManager (class, L81). That Java class owns the log-directory map — getOrCreateLog at L1239 maps a TopicPartition to its UnifiedLog — and background jobs for retention, flushing, and recovery that kick off in startup() at L771.

UnifiedLog (class, L105) is the per-partition commit log. Leader-side appends go through appendAsLeader (L1033), which validates record offsets, updates the leader epoch, and assigns contiguous offsets; follower-side replication enters via appendAsFollower (L1093). The write itself is delegated to LocalLog (class, L71), whose append at L586 writes into the active segment, and whose flush at L265 fsyncs up to a given offset — this flush boundary is what the recovery-point checkpoint tracks.

Each segment pairs a data file with two sparse indexes. LogSegment (class, L65) decides when to roll with shouldRoll (L167) — segment bytes reached, time rolled over, or index full — and appends records at L250, indexing the first record’s position per write call. The data file is FileRecords (class, L45, note the new record.internal package) backed by a memory-mapped file; append at L249 is just a write at the end, and truncateTo at L321 is what makes unclean leader elections recoverable. OffsetIndex (lookup, L97) maps relative offsets to file positions with binary search, and TimeIndex (class, L54) does the same for timestamps, enabling time-based retention and “read from yesterday 10:00” seeks. Idempotent and transactional producers are tracked by ProducerStateManager (class, L70), which persists sequence numbers as periodic snapshots (loadSnapshots, L184).

The high-watermark is the bridge between replication and visibility: UnifiedLog.highWatermark (L506) reports the largest offset replicated to all in-sync replicas, and the log only exposes records below it to consumers. Log-end sits at L2142; everything between watermark and log-end is written but not yet safe.

Clients: Producers and Consumers

KafkaProducer (class, L251) is a bookkeeping front end for a background machine. Both send overloads (L1027 and L1147) funnel into doSend (L1180), which serializes the key and value, waits up to max.block.ms for partition metadata, and appends the record into a ProducerBatch via tryAppend, L168. Batching is the producer’s main performance lever: linger.ms lets batches fill, batch.size caps them, and compression applies per batch. The Sender thread then drains and ships them, tracking in-flight batches for retries and sequencing.

KafkaConsumer (class, L552) is now a thin facade over two delegates. subscribe (L725) joins a group; poll (L987) simply forwards to whichever protocol you run. The classic delegate (ClassicKafkaConsumer, L117, poll at L657) uses the long-standing group protocol with heartbeats on a background thread; the newer AsyncKafkaConsumer (class, L178, poll at L936) implements the KIP-848 server-side rebalance protocol, moving most rebalance work out of a global lock. Both paths collect completed fetches into ordered records through FetchCollector (class, L51) and talk to the broker via ConsumerNetworkClient (send, L107), which wraps the shared NetworkClient, L89 — the same I/O core the producer uses, with send at L616 and a completion poll at L705. If you have ever wired Kafka into an Airflow DAG, this split matters: the consumer’s poll loop is where streaming and batch worlds hand off.

Group Coordination: The New Java Runtime

Consumer groups have historically been managed by a Scala coordinator embedded in the broker. On trunk that logic lives in a dedicated group-coordinator module. GroupCoordinatorService (class, L166) is the RPC surface — joinGroup at L1009, syncGroup at L1150, heartbeat at L1193, and commitOffsets at L2352 — all returning futures rather than blocking handler threads.

Internally the service routes each group to a shard keyed by the group’s partition in the coordinator’s log, and GroupCoordinatorShard (class, L158) is a deterministic state machine: membership changes and offset commits are first written as records, then applied to in-memory state, which makes group state reproducible after restarts — the same write-ahead-log philosophy as the log itself. Offset bookkeeping ends in OffsetMetadataManager (class, L80), whose commitOffset at L620 produces the records that back the well-known __consumer_offsets topic.

From Install to Your First Model

The README’s quickstart path is KRaft-only — no ZooKeeper. To build from source, run ./gradlew jar, then bootstrap a single-node cluster exactly as the README shows:

KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"
./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
./bin/kafka-server-start.sh config/server.properties

That is a real KRaft controller and broker in one JVM: the storage tool stamps a cluster ID into the metadata log, and the server starts from config/server.properties. From there the official quickstart at kafka.apache.org/quickstart walks you through creating a topic with bin/kafka-topics.sh and exercising the round trip with the console producer and consumer — a two-minute verification before you write any code.

For an AI workload, the natural next step is making your first event a model-relevant one: emit inference requests as JSON keyed by user ID, append feature updates from your training jobs, or ship GPU telemetry from every node. A Python producer (pip install kafka-python-ng or confluent-kafka) needs only bootstrap servers and a topic name to start appending; the durable log means a consumer that crashes loses nothing, it just resumes from its last committed offset. That replay property is what makes Kafka a good complement to Prometheus — metrics answer “how is the system now,” while the log answers “give me every event since last Tuesday.” And when the stream grows into a lake, the same events can be batch-drained to object storage or queried directly in ClickHouse for offline analysis.

Honest Limits

Kafka is not a database: there are no secondary indexes, no queries, no per-key updates — the OffsetIndex exists to reach a position in a file, nothing more. Retention is a blunt instrument configured in LogConfig (class, L65) — time- or size-based deletion of whole segments — so random access to old records is what compaction and external stores are for. Ordering is guaranteed only within a partition; choosing a good partition key for your events is a real design decision, and rekeying after the fact is painful. A broker JVM with modest defaults is fine to start with, but cross-AZ replication traffic, segment fsync tuning, and partition counts that now also drive KRaft metadata size all need attention as throughput grows.

The codebase itself carries its own friction: the ongoing migration from Scala core to Java modules means related logic is split across core, storage, and group-coordinator (the Scala LogManager even aliases the Java one), and two live consumer protocols mean any client-behavior reading must check which delegate is actually running. None of that diminishes the engineering here — it is just the price of a system that has been evolving this fast for this long, and reading the sources on trunk, as we did here, is the most reliable way to know what your cluster is really doing.

{“@context”: “https://schema.org”, “@type”: “TechArticle”, “headline”: “Kafka Under the Hood: How the Distributed Commit Log Actually Works”, “description”: “A source-level tour of Apache Kafka: broker request path, UnifiedLog storage engine, producer and consumer clients, and the Java group coordinator.”, “datePublished”: “2026-10-31”, “dateModified”: “2026-10-31”, “author”: “PyShine”, “publisher”: “PyShine”, “keywords”: “kafka, event streaming, distributed log, scala, java”, “url”: “https://pyshine.com/kafka/”}

Enjoyed this post? Never miss out on future posts by following us