Kafka Queues

by Marcus Databases System Design

🔍

# Kafka for System Design

### Prerequisites:
- Basic familiarity with queues and pub/sub

## Why do we need Kafka?

Suppose a payments service needs to notify a billing service every time a card is charged. The obvious design is a synchronous HTTP call. This works until any of the following happens:

1. **Rate mismatch**: payments spikes to 50k charges/sec, billing can only process 5k/sec. Requests time out and charges are lost.
2. **Availability coupling**: billing is redeployed, so payments starts failing too. One service's downtime became two.
3. **Fan-out**: fraud detection, analytics, and an audit log now *also* want to know about every charge. Payments must know about all four consumers and call each one.

The standard fix for (1) and (2) is a **queue**. Payments writes a message, billing reads it whenever it can, and neither needs the other to be up at the same moment.

But a traditional queue such as RabbitMQ or SQS does not fix (3), because a queue **deletes a message once it has been acknowledged**. The message is consumed exactly once, by one reader, and then it is gone. If four systems need the same event, you need four queues and a producer that knows about all of them. And if billing had a bug last Tuesday, the messages it processed incorrectly no longer exist, so there is nothing to reprocess.

We now ask: *what if the messages are persistent?*

## What is Kafka?

**Apache Kafka** is a distributed, durable, **append only log**. Producers append records to the end of the log, Kafka writes them to disk across a set of servers, and consumers read forward through the log at their own pace, each remembering its own position.

Every other property of Kafka follows from one decision:

---

**Kafka does not delete a record when it is read.** A record is removed only when the topic's retention policy expires it (7 days by default).

---

Because the log persists after reading, many independent readers can read the same records, and any reader can move backwards and read them again.

```
             payment-events,  partition 0   (one append only file on one broker)

offset      0         1         2         3         4         5
             ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
      record │  $12.00 │  $80.00 │  $ 4.50 │  $31.00 │  $ 7.25 │  $19.99 │◀── producer appends
             └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘    new records here
                  ▲                   ▲                   ▲
                  │                   │                   │
              retention         billing group      analytics group
              drops old          next reads          next reads
            records here          offset 2            offset 4
```

**Figure 1: A Kafka partition is an append only log.** There is one partition here and nothing else: a single file on a single machine. Each box is one record, and the number above it is that record's **offset**, its index in the file. Records are only ever appended at the right hand end, and a record only leaves at the left hand end, when retention expires it.

The arrows below the log are the readers' offsets. Neither of them removes anything; each just remembers how far it has read, so billing running two records behind analytics costs analytics nothing. Setting billing's offset back to 0 replays every record from the beginning.

Everything below is what Kafka adds to make the log fast, parallel, and able to survive a broker failure.

Some order of magnitude numbers, to see what the log buys us:

| Metric | Typical Value |
|---|---:|
| Sequential disk write (what the log does) | **~500 MB/s** |
| Random disk write (what a database index does) | **~1–10 MB/s** |
| Throughput per broker | **~100 MB/s to 1 GB/s** |
| Producer to consumer latency | **~5 ms** (2–10 ms) |
| Default retention | **7 days** |
| Default max message size | **1 MB** |
| Partitions per broker (practical) | **~1,000 to 4,000** |

The first two rows are the reason Kafka is fast despite writing everything to disk. Appending to the end of a file is sequential I/O, which is roughly two orders of magnitude faster than the random I/O a database performs, and Kafka never has to update a record in place.

## How does one log scale? Topics and partitions

A single log has two hard limits: it lives on one machine's disk, and it can be read in order by one reader. So Kafka splits it.

A **topic** is a *logical* name for a category of records, such as `payment events`. A **partition** is the *physical* log. A topic is made of one or more partitions, numbered from 0, and different partitions can live on different machines.

- Topics **organize** data
- Partitions **scale** data

The consequence is that **throughput and ordering are properties of the partition, not the topic**. Kafka guarantees that records within a single partition are read in the order they were written, and guarantees nothing about the relative order of records in different partitions, because those are separate files, written and read independently.

The **key** controls this tradeoff. The producer attaches a key to each record and Kafka hashes it to pick a partition:

```
partition = murmur2(key) mod num_partitions
```

It is worth being precise about what a key is, because this is a common misconception:

---

**A key is not a partition.** The mapping is many keys to one partition, never one to one. A partition is a file; a key is a label on a record that decides which file it goes into. With 10,000 accounts and 3 partitions, each partition holds roughly 3,300 different accounts, interleaved in the order they arrived.

---

What the key buys you is not a private log. It is the guarantee that all records sharing a key land in the *same* log, and are therefore read in the order they were written.

```
                             ┌────────────────────────────────────┐
                             │       TOPIC:  payment-events       │
                             │                                    │
                             │                                    │
                             │                                    │
┌────────────┐               │     Partition 0                    │
│  Producer  │──┐            │     ┌────┬────┬────┬────┐          │
└────────────┘  │            │     │ ▢  │ ▢  │ ▢  │ ▢  │          │─────┐         ┌───────────────┐
                │            │     └────┴────┴────┴────┘          │     ├────────▶│    Consumer   │
                │            │                                    │     │         │    Group 1    │
┌────────────┐  │ hash(key)  │     Partition 1                    │     │         └───────────────┘
│  Producer  │──┤──────────▶ │     ┌────┬────┬────┬────┐          │     │
└────────────┘  │  mod 3     │     │ ▢  │ ▢  │ ▢  │ ▢  │          │─────┤
                │            │     └────┴────┴────┴────┘          │     │
                │            │                                    │     │         ┌───────────────┐
┌────────────┐  │            │     Partition 2                    │     ├────────▶│    Consumer   │
│  Producer  │──┘            │     ┌────┬────┬────┬────┐          │     │         │    Group 2    │
└────────────┘               │     │ ▢  │ ▢  │ ▢  │ ▢  │          │─────┘         └───────────────┘
                             │     └────┴────┴────┴────┘          │
                             │                                    │
                             └────────────────────────────────────┘
```

**Figure 2: One topic, three partitions.** Zooming out one level from Figure 1: `payment-events` is not a file, it is a *name* for three files, and each of the three rows inside the box is a log exactly like the one in Figure 1.

On the left, every producer writes to the topic, and `hash(key) mod 3` decides which of the three logs each record lands in. Nobody picks a partition by hand. Since there are far more accounts than partitions, P0 is the log for `acct-17` *and* `acct-42` *and* `acct-88` and thousands more, interleaved in arrival order. Every `acct-17` charge lands in P0 and is read in order; an `acct-17` charge and an `acct-5` charge sit in different files and have no ordering relationship at all.

On the right, both consumer groups receive *every* record in the topic, because groups are independent and each tracks its own offsets. Within a group the partitions are divided up, which is what caps its parallelism at three; that split is Figure 4.

The box is a logical boundary, not a machine. Where these three files physically sit, and what happens when that machine dies, is Figure 3.

Two practical points follow:

1. **If no key is given**, the producer is free to place the record anywhere. Modern clients use *sticky partitioning*: they fill one batch for a partition, send it, then switch to another, which is better for batching than strict round robin. Either way, no useful ordering exists.
2. **A hot key is a hot partition.** If one merchant accounts for 40% of charges and the merchant ID is the key, one partition receives 40% of the traffic and one consumer does 40% of the work. Partitions balance keys, not traffic.

Note also that consumers **pull** from Kafka rather than having records pushed to them. This is a design choice: it stops a fast producer from overwhelming a slow consumer, and it is what allows each consumer to own its position in the log.

## What happens when a broker dies? The cluster

The topic box in Figure 2 is a logical boundary. Underneath it, those three files sit on real machines, and if they all sat on one machine that machine would be a single point of failure with a single disk and NIC as the ceiling. Zooming out one more level gives the real deployment.

A **broker** is one Kafka server; a **cluster** is a group of brokers. A partition's identity is the pair *(topic, partition index)*, and that pair, not the topic, is the unit the cluster distributes. There is no such thing as a broker that owns a topic. A single broker holds a mix of partitions belonging to many different topics, which is what the fourth row of each box below shows.

Each partition is replicated according to the topic's **replication factor**, the total number of copies kept. Exactly one copy is the **leader**, and it is the only copy that accepts writes. The others are **followers**, which continuously fetch from the leader.

```
                        Kafka cluster,  replication factor 3

Broker 1                     Broker 2                     Broker 3
┌────────────────────────┐   ┌────────────────────────┐   ┌────────────────────────┐
│ payment-events  P0   L │   │ payment-events  P0   F │   │ payment-events  P0   F │
│ payment-events  P1   F │   │ payment-events  P1   L │   │ payment-events  P1   F │
│ payment-events  P2   F │   │ payment-events  P2   F │   │ payment-events  P2   L │
│ user-signups    P0   F │   │ user-signups    P0   L │   │ user-signups    P0   F │
└────────────────────────┘   └────────────────────────┘   └────────────────────────┘
                                          ✕
                                   broker 2 dies
               payment-events P1 and user-signups P0 each elect a new
                 leader from an in-sync follower on broker 1 or 3;
                        every other partition is unaffected

L = leader,  F = follower
```

**Figure 3: The same three partitions, now spread across a cluster and replicated three times.** Three things to read off this figure. First, every partition exists on all three brokers, so losing a machine loses no data. Second, **leadership is spread**: each broker is the leader for one partition of `payment-events` and a follower for the other two, so writes to the topic are shared across all three machines instead of landing on one. Third, `user-signups` is interleaved with `payment-events` on the same brokers, because brokers hold partitions, not topics.

When broker 2 dies, only the partitions it led need an election, and only writes to those pause. Adding a fourth broker lets Kafka move partitions onto it and absorb more load.

Followers that are sufficiently caught up form the **in-sync replica set (ISR)**, and only an ISR member is eligible for promotion. Note that followers exist for **durability and failover, not read scaling**: reads are normally served by the leader, which prevents consumers from seeing records that have not yet been replicated. (Newer versions allow fetching from a follower in the same rack, but this is a network cost optimization, not a throughput one.)

The producer chooses how much durability it wants with the `acks` setting:

| `acks` | Behavior | Cost |
|---|---|---|
| `0` | Fire and forget | Fastest; records lost silently |
| `1` | Leader has written it | Lost if the leader dies before followers fetch |
| `all` | Every ISR member has it | Slowest; survives broker loss |

`acks=all` with replication factor 3 and `min.insync.replicas=2` is the standard "don't lose data" configuration.

## How is work divided? Consumer groups and rebalancing

A **consumer group** is a set of consumers sharing a group ID, which Kafka treats as one logical subscriber. Kafka assigns each partition of a subscribed topic to **exactly one** consumer in the group. The group as a whole therefore sees every record, but no two members process the same one.

Each consumer periodically **commits** the offset it has reached, so that after a restart it resumes from there instead of from the beginning.

The important consequence is that **the partition count is a hard ceiling on parallelism within a group**. A topic with 3 partitions and a group of 5 consumers has 3 consumers working and 2 sitting idle. Scaling consumption past that point means adding partitions, which is why partition count is a capacity decision made up front. A reasonable rule is to pick a partition count matching your *peak* expected consumer count with headroom, since adding partitions later changes which partition a key hashes to and breaks ordering for existing keys.

**Rebalancing** is the reassignment of partitions when group membership changes: a consumer is deployed, crashes, or stops sending heartbeats. Processing for the affected partitions pauses during a rebalance. Frequent rebalances are a common source of production latency, and the usual cause is not crashes but consumers taking too long between `poll()` calls and being declared dead while they are in fact still working.

Different consumer groups are fully independent: each has its own offsets, and each receives every record. That independence is what makes the next section possible.

## Queue or stream? Same infrastructure, different consumption

Kafka is often described as "a queue" or "a stream" as though these were different systems. They are not. The infrastructure is identical; what changes is **how many consumer groups read the topic**.

```
  QUEUE PATTERN                          STREAM PATTERN
  one group, work is divided             many groups, each sees everything

┌──────────────┐                       ┌──────────────┐ ─▶ billing group
  │ payment-     │ ─▶ billing group      │ payment-     │ ─▶ fraud group
  │ events       │     ├─ C1  (P0)       │ events       │ ─▶ analytics group
  │ P0 P1 P2     │     ├─ C2  (P1)       │ P0 P1 P2     │ ─▶ audit group
  └──────────────┘     └─ C3  (P2)       └──────────────┘
```

**Figure 4: The same topic read two ways.** On the left, one group splits the partitions among its members and each record is handled once; on the right, four independent groups each track their own offsets and each receive every record.

**Queue pattern.** One consumer group divides the partitions among its members and works through the backlog, each record handled by exactly one consumer. This is the pattern for background work: charging cards, sending emails, generating thumbnails. Use it for **async processing**, for **in-order processing per entity** when records are keyed, and for **decoupling producers from consumers** so neither side needs to know how many instances of the other exist.

**Stream pattern.** Several independent groups read the same records as they arrive. One charge event simultaneously feeds fraud detection, an analytics pipeline, and an audit log, and because the log persists, any of them can be replayed from any point. Use it for **real-time processing**, for **fan-out to multiple systems**, and wherever **replay** matters.

The dividing line: *if one pool of workers needs to split up a body of work, that is a queue; if several independent systems all need to see the same events, that is a stream.*

## What happens when a record fails to process?

Two failure modes matter, and they have different answers.

**1. A record cannot be processed.** Because a consumer advances through a partition in order, a bad record blocks everything behind it. A consumer that keeps retrying never commits a higher offset, and the partition stalls indefinitely. Head-of-line blocking here is per-partition, not per-topic, but a stalled partition is still a stalled tenth of your traffic.

The standard solution is a **Dead Letter Queue (DLQ)**. After a bounded number of retries, the consumer publishes the failing record to a separate topic, commits its offset, and moves on. The DLQ is then inspected, fixed, and replayed without holding up live traffic.

**2. Records were processed, but incorrectly.** If a bug caused a group to compute the wrong thing, the group's offsets can be reset to an earlier point and the records reprocessed once the fix ships. This is only possible because Kafka retains records after reading, and it is the main practical advantage over a traditional queue.

Both rest on a delivery guarantee:

---

**Kafka is at-least-once by default.** A consumer can process a record and then crash before committing its offset, so on restart it processes that record again.

---

The consequence is that **consumers must be idempotent**. Charging a card on every delivery is a bug; charging against a payment ID that is checked first is not. Kafka does offer exactly-once semantics via idempotent producers and transactions, but this only covers Kafka-to-Kafka pipelines and costs throughput. If a consumer writes to an external system, idempotency is your responsibility, not Kafka's.

## When is Kafka the wrong tool?

Reaching for Kafka by default could be a mistake. It is the wrong choice when:

1. **You need per-message acknowledgement or deletion.** SQS and RabbitMQ let a consumer ack, nack, or extend a visibility timeout on a single message. Kafka only tracks a single advancing offset per partition, so "process these 10,000 tasks in whatever order, retry the ones that fail" is awkward.
2. **You need priority queues or per-message delays.** Kafka has no notion of either. A log has one order, the one records were written in.
3. **You need very high fan-out to many independent consumers with tiny volume.** A managed pub/sub service is less to operate.
4. **The volume is small.** A three-broker cluster with ZooKeeper or KRaft, monitoring, and partition planning is real operational cost. At a few hundred messages/sec with one consumer, SQS could be the correct answer.

The signals that Kafka *is* right: 
- High throughput
- Multiple independent consumers of the same data 
- Need to replay
- Per-entity ordering

## Putting it together

The standard production topology is:

*A topic partitioned by an entity key, replication factor 3 with `acks=all` and `min.insync.replicas=2`, partitions spread across brokers with leadership balanced, and one consumer group per downstream system, each with idempotent consumers and a DLQ.*

The two sizing dimensions are independent, because they solve different problems:

1. **Partition count** is driven by *throughput and parallelism*: peak records/sec divided by what one consumer can process, with headroom, since increasing it later breaks key-to-partition stability.
2. **Replication factor** is driven by *durability*: 3 is the near-universal default, tolerating one broker loss with `min.insync.replicas=2`.

### System Properties and Constraints

| System concern | Behavior and constraint |
|---|---|
| Storage | Append-only sequential writes make it fast and replayable, but records are immutable and there is no random access by content, only by offset |
| Ordering | Guaranteed within a partition, so keying by entity gives per-entity ordering; across partitions there is no order at all |
| Parallelism | Partitions are the unit of concurrency, so consumers scale to the partition count and no further; extra consumers idle |
| Partition count | Set up front for peak load; raising it later rehashes keys to different partitions and breaks ordering for existing keys |
| Key distribution | Hashing balances keys evenly, but a hot key still concentrates traffic on one partition and one consumer |
| Node failure | An in-sync follower is promoted in seconds; writes to that partition fail during the election, so producers need retries |
| Durability | `acks=all` survives broker loss but adds latency; `acks=1` is fast and loses data if the leader dies before followers fetch |
| Delivery | At-least-once by default, so consumers must be idempotent; exactly-once exists but only within Kafka and at a throughput cost |
| Failure isolation | A poison record blocks only its own partition, and a DLQ unblocks it, at the cost of that record now being out of order |
| Retention | Retaining after read enables replay and multiple readers, at the cost of disk proportional to throughput times retention window |
| Consumer liveness | Heartbeats detect dead consumers, but a slow consumer looks identical to a dead one and triggers a rebalance that pauses processing |

## Definitions Cheat Sheet

| Term | Definition |
|---|---|
| **Broker** | One Kafka server; stores partitions and serves producers and consumers |
| **Cluster** | A group of brokers working together |
| **Topic** | A logical category of records, and the unit consumers subscribe to |
| **Partition** | One ordered, physical log within a topic; the unit of ordering, parallelism, and replication |
| **Offset** | A record's position within a partition; consumers commit offsets to save progress and can reset them to replay |
| **Key** | A value attached to a record and hashed to select a partition; not unique, and shared by all records that must stay ordered together |
| **Consumer group** | Consumers sharing a group ID, treated as one logical subscriber; each partition is assigned to exactly one member |
| **Leader replica** | The only copy of a partition that accepts writes, and normally the one serving reads |
| **Follower replica** | A copy that fetches from the leader; exists for durability and failover, *not* read scaling |
| **ISR** | In-sync replica set: the replicas caught up enough to be promoted |
| **Replication factor** | Total number of copies of each partition |
| **Rebalancing** | Reassigning partitions when a consumer joins, leaves, or fails; pauses processing on affected partitions |
| **DLQ** | A separate topic for records that repeatedly fail processing, so they stop blocking their partition |