How Kafka Works: Topics, Partitions, and Consumer Groups

The messaging articles pre-paid this one’s debts in writing. Message queues deferred the log-and-partition model here (ordering within a partition, not across them) deferred the transactional producer’s mechanics here, and named this stack “a durable log built for replay and high-throughput streaming: consumers track offsets, and history stays available.” Event-driven architecture leaned on the same substrate (“a log-backed topic keeps history, so a subscriber that fell behind, crashed, or was deployed late catches up by replaying it”) and asked this article to show how retention, partitions, and consumer groups let one topic serve replay, worklists, and broadcasts at once. That is the promise this article pays, in that order.

Kafka architecture is a distributed log: producers append records to the ends of partitioned, replicated logs; the logs retain their records for a configurable window; and consumers read at their own pace, tracking a position (an offset) in each partition and committing that position as they go. The log, not a queue’s transit, is the core abstraction, and every behavior the messaging articles admired (replay, independent consumption, durable history) is a consequence of the choice.

The durable log: the core abstraction

A log is an append-only, ordered sequence of records, addressed by position, and choosing it over a queue’s transit is the design decision everything else follows from. In a classic queue, a message is a parcel: delivered, acknowledged, gone. In a log, a record is a line in history: written once, kept for a retention window, readable by anyone with the position to read from. Consumers are readers with bookmarks, not recipients with receipts; the broker does not track who received what; the consumers track themselves. The same immutability choice the search stack made for its segments, made here for a stream: once written, a record never changes, and concurrency stops being a problem.

Why the model scales deserves its own sentence, because it is the reason the largest pipelines in the industry run on logs. The broker’s work per record is constant regardless of how many consumers exist: no per-recipient state, no per-recipient delivery, no bookkeeping that grows with the audience; fan-out is consumers reading, not the broker delivering. Adding a reader costs the log nothing, and the sequential, batched appends the log’s shape encourages are exactly what storage systems do fastest. The economics follow the shape.

What the choice buys is the list event-driven architecture admired: decoupled consumption, with any number of readers at any number of paces, because the log does not care who is behind; replay, because a subscriber that crashed or shipped late seeks back to where it left off and reads forward: history, not re-delivery; and debuggability, because an incident becomes “rewind the tape and watch,” the same forensic luxury a write-ahead log gives a database. What it costs is storage for the window and an ordering that is only as global as a partition, the boundary the next section draws precisely.

The retention honesty belongs up front, in the exact words the event article used: retention is a cleanup policy, not a storage contract. The log keeps a bounded window (by time, by size, or, for keyed records, by compacting to the latest value per key) and a system that treats the window as its system of record has moved its database into a cleanup policy without noticing. The transport-and-history window is the design; the truth stays where the architecture put it.

Topics and partitions: the units of order and parallelism

A topic is the named stream, the subscription boundary producers write to and consumers read from. Under it, the topic divides into partitions: independent logs, each strictly ordered, each growing by append. This is where the ordering promise the queue article deferred gets exact: ordering holds within a partition, never across partitions; the producer chooses a partition per record, hashed from the record’s key when one exists, so the same entity’s records land in the same partition in write order, while records without keys spread for throughput and carry no cross-partition promise. Per-entity ordering without global ordering is the whole deal, and it is the data partitioning concept applied to a stream: the key is the partition key.

The partition is also the parallelism unit and the scaling unit. One consumer per partition within a group is the ceiling of that group’s throughput; more partitions mean more parallel writers, more parallel readers, and more inter-partition fan-out for anything that needs the whole topic. The hot-key problem arrives here too (one busy entity hashes to one partition, and no amount of fleet growth spreads it) and the partition count is fixed at topic creation for the same reason every hash-based placement is: change the modulus, and every key’s home changes, breaking the per-entity order the partitioning exists to keep.

The partition count is therefore the topic’s most consequential early decision, and the honest guidance is arithmetic, not superstition. Enough partitions to serve the expected consumer concurrency (the group’s ceiling is one member per partition) and to spread the write rate, with headroom for growth because the count is fixed; not so many that every reader pays assignment overhead and the fleet idles beneath tiny shards. Teams that pick a large number “to be safe” buy rebalancing churn and storage fragmentation; teams that pick one buy a throughput ceiling they will meet at the worst time. The number comes from the measured write rate and the planned consumer fleet, revisited by re-creation when reality disagrees; the same fixed-count honesty every hash placement in this project has paid.

Each partition is replicated: a leader handles the partition’s writes, followers tail the leader, and a failed leader is replaced from the followers that are caught up: the leader-follower mechanics replication teaches, running per partition. The result is the batch’s recurring architecture one more time: partitioned for parallelism, replicated for survival: the search stack’s shape, on a different payload.

Consumer groups and offset management

A consumer group turns a broadcast into a worklist. The group’s members split the topic’s partitions (each partition assigned to exactly one member) and together they drain the stream, each record reaching one member once. The group’s throughput is bounded by its partition count: more members than partitions means idle members; more partitions than members means each member holds several and processes them in turn. This is the unit of horizontal scale on the read side; the partitioning decision made at write time becomes the parallelism ceiling at read time, and the ceiling is chosen once, up front.

Because a second group on the same topic gets its own assignment of the same partitions, the one-topic-many-shapes promise lands here: a topic with a billing group, a notification group, and an analytics group is a broadcast (every record reaches every group) while inside each group it is a worklist; every record reaches exactly one member. And any group can rewind, because the log is still there. That is the triple the event article promised this stack for: worklists, broadcasts, and replay, from one substrate, chosen per consumer rather than per broker.

Offset management is where delivery guarantees become a dial. A consumer’s position in each partition is an offset, and the group periodically commits its positions (its bookmarks) back to the log system. The timing of the commit is the entire guarantee: commit after processing, and a crashed member’s partitions return to the last committed offset and reprocess, at-least-once, the queue article’s default and the reason consumers need idempotency on their side; commit before processing, and a crash skips records, at-most-once, chosen only where a gap beats a duplicate. The offsets themselves are durable and checkpointed, so a member that dies hands its partitions to a survivor at the committed line, and the stream’s history makes the recovery a rewind rather than a mystery.

Rebalancing is the group’s operational reality. Membership changes (a member joins, dies, or is deemed dead by missed heartbeats) and the partitions redistribute among the survivors. The redistribution pauses the group’s consumption for its duration, the stop-the-world moment the older protocol made expensive; modern incremental rebalancing shrinks the pause by moving only the partitions that must move. The failure pattern to design against is the storm: a member that misses heartbeats under load triggers rebalances that pause everyone, and the load spike that caused the missed heartbeat returns worse; the fix is sizing, timeouts tuned to reality, and the honest monitoring of rebalance frequency as a first-class metric.

Delivery, durability, and the transactional producer

The producer’s side of the durability question is a three-way setting: acknowledge the write immediately, when the partition leader accepts it, or when the leader and its caught-up followers have all persisted it. The third setting is the durability one (it makes an acknowledged record survive a leader failure by construction) and it is priced in latency, the same write-path waiting game every replicated system plays. Retries complete the picture: a producer that times out and resends can duplicate records, which is why the broker-side deduplication of the idempotent producer exists; sequence tracking per producer so a retry lands as the same record it tried to be. Delivery is built from these parts: at-least-once underneath, made effectively-once by machinery on either side.

The transactional producer is the machinery the queue article pointed at for the hard case. A pipeline member that consumes from one topic, transforms, and produces to others can commit, in one atomic step, the records it produced and the offsets of what it consumed, so a crash mid-transform either leaves none of its work visible or all of it. This is the real exactly-once of the messaging world: not a delivery property, but a commit property, assembled from the log’s offsets and a coordinator that marks transactions. The decision rule the queue article set still holds, pay for it where duplicates cost money and resist dedup; use idempotent consumers everywhere else, and this section is the machinery that makes the first option real.

The honest fine print on the durability side is the in-sync set. A partition’s followers fall behind under load; a follower that falls too far behind leaves the in-sync set, and the leader’s durability waits only for the followers that remain. Acknowledge-after-all therefore means “all who are caught up,” and the cluster’s configuration decides how quickly a lagging follower is dropped: the same trade read/write separation prices on the database side between waiting and falling behind, applied here to write acknowledgment. Durability is a number of copies in agreement, and that number is a monitored fact, not a static promise.

Retention, compaction, and replay

Retention keeps the window of history. The default is time (a week is the conventional starting point) with size as the other bound, and the choice sets the promise the stream makes to its consumers: how far back a late subscriber or a rebuilt service can rewind. Retention is the subscription’s memory, and the sizing question is honest about who needs how much of it: analytics groups that rebuild weekly need a deep window; a transient worklist can run shallow, because a group that never rewinds past a day is paying storage for history nobody reads.

Compaction is the special case that makes the log a store of current state. For topics keyed by entity, the log can be compacted (keeping the latest record per key and discarding the earlier ones) so the topic becomes a changelog: a replay from the beginning yields the current value of every key, and the storage stays bounded. This is the mechanism behind the compacted topics that back restore-patterns and cache-rebuilds, and it deserves the same warning the event article attached: a compacted topic of latest-values is a derived copy, not a system of record; useful precisely because it can be rebuilt, dangerous the moment it is treated as truth.

Replay is the dividend all this pays, and the operational patterns deserve their list. A service deployed for the first time builds its state by reading the topic from the beginning. A service with a corrupted cache resets and rebuilds, the projection-rebuild escape the event article named. An incident investigation rewinds the relevant partition and watches the records cross in order, the forensic read the debuggability promise made. And a backfill of a downstream store is a consumer group pointed at an old offset, running forward at full speed. None of these require anything of the producers; the history was already the design.

FAQ

How does Kafka work?

As a set of partitioned, replicated logs. Producers append records to topic partitions, chosen by hashing a key or spreading for throughput; the broker stores each partition as an ordered, retained log with a leader and followers per partition; and consumer groups read the logs at their own pace, committing offsets as their bookmarks. Everything distinctive (replay, independent consumers, per-entity ordering) follows from the log being the core abstraction rather than a queue’s transit.

What are Kafka topics and partitions?

A topic is the named stream; partitions are its independent, ordered logs. The partition is the unit of ordering (the same key always lands in the same partition, in write order) and the unit of parallelism: one consumer per partition per group, more partitions for more scale, the count fixed at creation because changing it would rehome every key.

What is a consumer group?

A set of consumers that splits a topic’s partitions among its members, so the group reads like one logical subscriber while each record reaches exactly one member, a worklist. Multiple groups on the same topic each get the full stream, a broadcast. One substrate, both shapes, chosen per consumer.

What are offsets, and how are they managed?

An offset is a consumer’s position in a partition’s log. The group commits its offsets as it processes, and the commit’s timing is the delivery guarantee: after processing for at-least-once, before for at-most-once, with idempotent consumers making the repeats safe. Because positions are durable, a crashed member’s partitions resume from the last committed line.

How long does Kafka keep messages?

For the retention window (by time or size, per topic) or, on compacted topics, the latest record per key indefinitely. The window is the replay promise: as far back as a late or rebuilding consumer needs to rewind. It is a cleanup policy, not a storage contract; the history window is the design, and the system of record stays wherever the architecture put it.

Related articles

• Next read: kafka vs rabbitmq; the broker decision this architecture makes concrete: log versus transit, replay versus routing, and when each model wins.
• message queues, the fundamentals: delivery guarantees, ordering, dead letters, the contracts this machinery implements.
• event-driven architecture; the style this substrate carries: pub/sub, projections, and the rebuild-from-log escape.
• data partitioning, the concept underneath partitions: rule, placement, skew, and the fixed-count arithmetic.
• database replication; the leader-and-follower mechanics each partition runs, and the catch-up trade behind acks. The partition is the replication unit; the broker is a fleet of leaders, each with its own followers and its own lag.
• idempotency; the consumer-side half of at-least-once: dedup keys and the discipline that makes repeats safe. The log keeps every record forever; the consumer decides which of them count, and how to prove it.
• backpressure, the slow-consumer math for the groups that cannot keep up with their partitions.

Last updated on 21 September 2026.