Scenario 03.3.2 · Rack rebalance

Kafka rack-aware reassignment after adding a rack

Replication · Advanced · about 3 min · Apache Kafka 4.3

With only 2 racks, RF=3 must put 2 replicas in one rack; add a 3rd rack and rack-aware reassignment spreads them.

What you start with

With only two racks, a 3-replica partition must place two copies in one rack, leaving it exposed if that rack fails. Here a third rack is added and each partition is reassigned rack-aware so its replicas spread across all three.

Step by step

1. Cluster ready t+0.000s Four brokers across 2 racks (rack-a × 2, rack-b × 2). With RF=3 and only two racks, every partition must put 2 replicas in one rack. Open at this step: step 1, Cluster ready
2. Record O1 committed t+6.000s O1 acknowledged by the leader (acks=all). All 3 ISR replicas in sync. Open at this step: step 2, Record O1 committed
3. broker-5 joined rack-c t+9.000s A fifth broker joins in rack-c. No partitions yet - each partition still clumps 2 replicas in one rack. Open at this step: step 3, broker-5 joined rack-c
4. Partition 0 reassigning t+12.050s reassign_rack_aware fires for partition 0: one rack-a or rack-b replica is replaced by broker-5 (rack-c). New replica is catching up. Open at this step: step 4, Partition 0 reassigning
5. Partition 0 rebalanced t+12.600s Partition 0 ISR reconverged. Replicas now span rack-a, rack-b, and rack-c. Open at this step: step 5, Partition 0 rebalanced
6. Partition 1 reassigning t+16.050s reassign_rack_aware fires for partition 1. Open at this step: step 6, Partition 1 reassigning
7. Partition 1 rebalanced t+16.600s Partition 1 ISR reconverged across all 3 racks. Open at this step: step 7, Partition 1 rebalanced
8. Partition 2 reassigning t+20.050s reassign_rack_aware fires for partition 2. Open at this step: step 8, Partition 2 reassigning
9. All partitions rebalanced t+20.600s Every orders partition now has one replica in each of rack-a, rack-b, and rack-c. ISR is full on all partitions. Open at this step: step 9, All partitions rebalanced

The takeaway

Adding capacity does not automatically rebalance existing partitions - a rack-aware reassignment is what redistributes replicas to restore per-rack fault tolerance.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.3.2" title="Kafka Simulator: Rack rebalance" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/rack-rebalance/">Kafka rack-aware reassignment after adding a rack</a></p>

Further reading

Apache Kafka documentation

More in Replication

• Kafka ISR (in-sync replicas): how replicas leave and rejoin
• min.insync.replicas with acks=all: when Kafka refuses writes
• Kafka slow follower: replica.lag.time.max.ms and the ISR
• Kafka high watermark pinned by a slow in-sync replica
• Losing two Kafka brokers with min.insync.replicas=2
• Kafka preferred leader election after a rack failure
• Kafka unclean.leader.election.enable: offline or data loss
• Kafka unclean leader election: permanent loss of acked data
• Kafka log truncation when an old leader returns
• Kafka broker.rack: rack-aware replica placement
• Increase the replication factor of an existing Kafka topic

All scenarios