Redis Clusters

by Marcus Databases System Design

🔍

### Prerequisites: 
- [Redis for System Design](https://finetuneinterview.com/lessons/redis-for-system-design)

## Why do we need several Redis instances?

Recall that Redis (REmote DIctionary Server) is an in memory (RAM) data structure that is used as a fast database, for primarily key-value queries. This data structure is extremely **fast** and is used heavily for applications that require high availability and low latency, such as caching.

To motivate this, these are some of Redis' performance numbers:

| Metric | Typical Value |
|---|---:|
| Memory access (RAM) | **~0.0001 ms** |
| Key lookup (`GET`) | **O(1)**, **~0.1 ms** (0.05–0.3 ms) |
| Key write (`SET`) | **O(1)**, **~0.1 ms** (0.05–0.3 ms) |
| **End-to-end request (`GET` over network)** | **~1 ms** (0.5–2 ms) |
| Throughput (single node) | **~1 million ops/sec** (500K–2M ops/sec) |

As we can see Redis is very fast, practically, if we were to request an entry within Redis, we would receive this within about 1 ms, which is faster than a database lookup.

However, we now ask, *Why can't we just use only 1 Redis instance for our service?*
1. Memory: One Redis instance is constrained by the size of RAM on its host machine
2. Throughput: Each Redis instance is *single threaded* for command execution, so we get a maximum throughput per node of about **1 million ops/sec**.

To solve these issues, we can introduce **Redis Clusters** which enable use to scale using several Redis Instances.

## What's a Redis Cluster?

A **Redis Cluster** is the official way to run Redis across multiple machines. It solves both of our problems at once:

1. **Memory**: the dataset is *sharded* (partitioned) across several primary nodes, so total capacity is the sum of all nodes' RAM
2. **Throughput**: reads and writes are spread across nodes, so total ops/sec scales roughly linearly with the number of primaries

The key idea behind Redis Cluster is the **hash slot**.

### Hash Slots

The entire keyspace is divided into exactly **16,384 slots**. Every key deterministically maps to one slot:

```
slot = CRC16(key) mod 16384
```

Each primary node in the cluster *owns* a contiguous range of slots. For example, with 3 primaries:

| Node | Slot Range | Share of Keyspace |
|---|---|---:|
| Primary A | 0 to 5460 | ~33% |
| Primary B | 5461 to 10922 | ~33% |
| Primary C | 10923 to 16383 | ~33% |

When a client runs `SET user:123 "..."`, it computes `CRC16("user:123") mod 16384`, gets (say) slot **8412**, and sends the command to **Primary B**, because B owns that slot. That's the entire routing model: *key → slot → node*.

Each primary also has one or more **replicas** that copy its data asynchronously. Replicas serve two purposes:

1. **High availability**: if a primary dies, one of its replicas is promoted and takes over its slot range
2. **Read scaling**: clients can optionally read from replicas (accepting slightly stale data)

So a production 3-shard cluster looks like this:

```
                        ┌──────────────────────┐
                        │        Client        │
                        │ CRC16(key) mod 16384 │
                        └──────────┬───────────┘
              ┌────────────────────┼────────────────────┐
              ▼                    ▼                    ▼
     ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
     │   Primary A    │  │   Primary B    │  │   Primary C    │
     │ slots 0 to 5460│  │ 5461 to 10922  │  │ 10923 to 16383 │
     └────────┬───────┘  └────────┬───────┘  └────────┬───────┘
              │ async             │ async             │ async
              ▼                   ▼                   ▼
     ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
     │   Replica A    │  │   Replica B    │  │   Replica C    │
     └────────────────┘  └────────────────┘  └────────────────┘
```

Notice that **sharding** and **replication** are two *independent* dimensions here: sharding scales memory and write throughput, replication provides failover and read throughput.

## How does routing actually work?

Here's the part that surprises most people: **there is no central router or proxy** in a Redis Cluster.

Instead, routing is fully decentralized:

1. Every node knows the complete slot → node mapping
2. **Cluster-aware clients** (Jedis, redis-py, ioredis, etc.) fetch that map on startup and cache it locally
3. If a client sends a command to the *wrong* node, the node replies with a redirect:

```
MOVED 8412 10.0.0.2:6379
```

The client then updates its cached map and retries against the correct node. In steady state, redirects almost never happen: the client hits the right node on the first try.

In production, because routing is decentralized, **there is no single point of failure at the routing layer**. Every application server carries its own slot map, so routing capacity scales automatically with your application fleet: adding more app servers adds more routing capacity for free. Compare this to proxy-based designs, where the proxy tier is a separate piece of infrastructure you must provision, monitor, scale, and fail over. It adds a network hop to every request, and if it goes down, your entire cache is unreachable even though every Redis node is healthy.

## How does failover work?

Redis Cluster nodes constantly gossip with each other over a separate **cluster bus** port (the data port + 10000). Each node pings a few random nodes every second, and health information spreads through the cluster.

Failover works like an election:

1. If a **majority of primaries** agree that a primary is unreachable, that primary is marked as failed
2. One of its replicas wins an election (the one with the most up-to-date data is favored)
3. The winning replica promotes itself to primary and **takes over the failed node's slot range**
4. The new topology gossips through the cluster, and clients update their slot maps via `MOVED` redirects

The key property: this happens **automatically, with no external coordinator**. (The older *Redis Sentinel* system achieves failover for non-sharded Redis, but requires running separate watcher processes. Cluster builds this in.)

Note the phrase *majority of primaries*: this is why a production cluster needs **at least 3 primaries**. With fewer, there's no meaningful quorum, and the cluster can't safely distinguish "that node died" from "I'm the one who got partitioned away."

## What breaks when you shard? (Multi-Key Operations)

Sharding is not free. Because keys now live on different machines, any command touching **multiple keys** (`MGET`, `SUNION`, transactions with `MULTI`/`EXEC`, Lua scripts) only works if *all keys hash to the same slot*.

The escape hatch is **hash tags**: if a key contains a `{...}` section, *only the part inside the braces* is hashed.

```
user:{123}:profile   → CRC16("123") → slot 7365
user:{123}:sessions  → CRC16("123") → slot 7365   (same slot!)
user:{123}:cart      → CRC16("123") → slot 7365   (same slot!)
```

All three keys are guaranteed to land on the same node, so you can `MGET` them or run a transaction across them. The trade-off: you're deliberately *defeating* the even distribution for those keys, so if one user is extremely hot, their entire key group hits one shard.

## Why not Consistent Hashing?

If you've studied distributed caches, you might be wondering: *isn't this exactly the problem consistent hashing solves?* Why did Redis invent hash slots instead of using the classic hash ring?

The funny answer: **hash slots basically *are* consistent hashing, with the randomness removed.** Both solve the identical problem: *when a node joins or leaves, move only a small fraction of keys instead of rehashing everything.* They just implement it differently:

| | Consistent Hashing (Ring) | Redis Hash Slots (Table) |
|---|---|---|
| Keyspace chunks | Arcs between node points | 16,384 numbered slots |
| Chunk assignment | Falls out of the hash function (random) | Explicit table (exact) |
| Balance | Statistical, needs virtual nodes | Exact by construction |
| Unit of migration | Implicit hash range | Named, enumerable slot |
| Operator control | Hard, you fight the math | Easy, just edit the table |
| Coordination required | None | Gossip (which Redis has anyway) |

Let's unpack why Redis chose the table.

**1. The ring's balance is only statistical.** Place 3 nodes at random points on a ring, and the arcs are almost certainly *not* equal: one node might own 50% of the keyspace by bad luck. The classic fix is **virtual nodes**: each physical server gets 100 to 200 points on the ring so the law of large numbers evens things out. But notice what virtual nodes really are: *pre-partitioning the keyspace into many small chunks and assigning chunks to nodes*, which is exactly what hash slots do, deterministically instead of randomly. (Cassandra's *vnodes* and Kafka's *partitions* are the same pattern.)

**2. Slots are a clean unit of migration.** Resharding happens slot by slot. A slot is a small, *named, enumerable* thing: you can literally ask "which keys live in slot 8412?" (`CLUSTER GETKEYSINSLOT`) and track migration progress precisely. On a ring, the unit of movement is "the arc between two hash values", an implicit range you can't enumerate or checkpoint as cleanly.

**3. Operator control.** Because the slot → node mapping is just a table, you can edit it deliberately: give a beefier machine 2× the slots, move a *hot* slot off an overloaded node, or drain a node gracefully before decommissioning. A hash ring's assignment falls out of the hash function; steering it means fighting the math.

**4. The table is tiny.** 16,384 slots fit in a **2KB bitmap** (16,384 bits), small enough to ride along inside the regular gossip heartbeat packets. This is literally why the number is 16,384 and not 65,536: antirez (Redis's creator) has explained that 65k slots would bloat the ping messages, while 16k is plenty for the practical ceiling of ~1,000 nodes.

**5. The ring's superpower is wasted here.** Consistent hashing shines in *coordination-free* settings, such as classic client-side memcached sharding or Dynamo-style databases, where any client can compute key placement independently with zero shared state. Redis Cluster already pays for coordination (gossip, config epochs, elections), so maintaining one small shared table costs nothing extra, and you get all the control benefits above for free.

**Note:** consistent hashing and fixed hash slots are two implementations of the same idea, *minimal data movement on topology change*. The ring optimizes for decentralization with zero coordination; fixed partitioning optimizes for exact balance, controllable migration, and operational visibility. Most modern systems (Redis, Cassandra, Kafka) converged on pre-partitioning; the pure ring survives mostly in client-side memcached sharding.

## Resharding: Adding a node without downtime

Suppose our 3-shard cluster is running out of memory and we add a **Primary D**. What happens?

Slots get migrated from the existing nodes to D, *live, while serving traffic*:

1. A slot is marked **`MIGRATING`** on the source node and **`IMPORTING`** on the target node
2. Keys in that slot are moved over one batch at a time (`MIGRATE` command)
3. During the transition, if a client asks the source node for a key that has *already moved*, it gets a one-time **`ASK`** redirect to the target node (unlike `MOVED`, `ASK` doesn't update the client's slot map, since the migration isn't finished yet)
4. Once the slot is empty, ownership officially flips, and clients learn the new mapping via `MOVED`

Repeat for however many slots D should own (e.g., ~4,096 slots for an even 4-way split). At no point does the cluster stop serving requests. This is the payoff of slots being explicit, enumerable units: migration is trackable, resumable, and incremental.

## The consistency caveat

Redis Cluster is **not strongly consistent**, however it is highly available.

Replication from primary to replica is **asynchronous**. A primary can acknowledge a write to the client and then die *before* the replica received that write. After failover, the promoted replica simply doesn't have it: **the acknowledged write is lost**.

Whether this matters depends entirely on your use case:

1. **As a cache**: this is fine. The source of truth is your database; a lost cache entry is just a cache miss.
2. **As a primary datastore**: this is a real trade-off you must name. If losing acknowledged writes is unacceptable, Redis Cluster alone is the wrong tool (or you need `WAIT` semantics and to accept the latency cost, and even then guarantees are weaker than a consensus-based store).

## How this appears in real systems

In practice, very few teams run raw Redis Cluster themselves. The common setups:

1. **Managed offerings**: AWS ElastiCache, Google Memorystore, Azure Cache, Redis Enterprise. These run cluster mode for you and handle failover and resharding behind an endpoint. This is what "we'll use a distributed Redis cache" means at most companies.
2. **Client-side sharding**: the DIY version, where application code hashes keys (often with consistent hashing) and picks a node itself. Simple, but resharding and failover are *your* problem.
3. **Proxy-based sharding**: Twemproxy, Envoy, or similar in front of plain Redis instances, so clients stay dumb. Costs an extra network hop and another component to operate.

## Putting it together: The production topology

The standard production deployment is:

*N shards, each shard being a primary plus one or two replicas, with keys distributed by hash slot (CRC16 mod 16384) and replicas serving both failover and read scaling.*

The two dimensions are sized **independently**, because they solve different problems:

1. **Shard count** is driven by *memory and write throughput*: total dataset size divided by usable RAM per node, and total write ops/sec divided by per-node capacity, whichever demands more shards
2. **Replica count** is driven by *availability and read throughput*: at least one replica per shard for failover, more if replicas absorb read traffic

### System Properties and Constraints

Every design choice in Redis Cluster both provides us with a pro and a con (a constraint). This is the summary to internalize:

| System concern | Behavior and constraint |
|---|---|
| Routing | Smart clients cache the slot map and go direct to the owning node; no router tier to operate, but every client library must be cluster-aware |
| Node failure | Gossip detects it, a majority of primaries agree, and a replica is promoted; failover takes seconds, and writes to that shard fail during the window, so applications need retry logic |
| Multi-key operations | Only work within a single slot; hash tags `{...}` co-locate related keys, at the cost of concentrating a hot user's traffic on one shard |
| Adding capacity | Live slot migration with `MIGRATING`/`IMPORTING` states and `ASK` redirects; no downtime, but migration consumes CPU and network on both nodes, so it is throttled and run off-peak |
| Consistency | Async replication means acknowledged writes can be lost on failover; acceptable for a cache backed by a database, a serious constraint if Redis is the source of truth |
| Cluster size | 16,384 slots as a 2KB bitmap keeps gossip cheap, with a practical ceiling around 1,000 nodes; beyond that, gossip traffic itself becomes the bottleneck |
| Key distribution | Explicit slot assignment gives exact balance and operator control, but a skewed *access* pattern (one hot key) still overloads a single shard; slots balance keys, not traffic |

Notice the pattern in the right column: every row is a *feature paired with its cost*.