Scenario 08.0.6 · Correlated failure: a whole rack goes down
Kafka rack failure: why RF=4 on two racks refuses writes
Replicas spread over two racks; one rack fails as a unit, taking the leader and a follower at once. The survivors hold the data but the ISR drops below min.ISR - RF alone is not enough.
Run this scenario in the playgroundHealthy cluster (4 replicas, 2 racks). t+1.000s
RF=4, min.ISR=3. broker-1,2 sit in rack-a; broker-3,4 in rack-b. broker-1 leads.
Open at this step: step 1, Healthy cluster (4 replicas, 2 racks)O1,O2 committed, HW=2. t+5.000s
O1,O2 commit on all four replicas (HW=2).
Open at this step: step 2, O1,O2 committed, HW=2rack-a down → ISR 2 (below floor). t+9.000s
A whole failure domain - rack-a - goes down at once, taking the leader broker-1 AND broker-2 together. The survivors {broker-3, broker-4} in rack-b elect broker-3 (epoch→1) and still hold every committed record, but the ISR is now 2, below min.ISR=3.
Open at this step: step 3, rack-a down → ISR 2 (below floor)O3 rejected: NOT_ENOUGH_REPLICAS. t+12.000s
O3 is refused with NOT_ENOUGH_REPLICAS - a single correlated rack failure breached the durability floor even at RF=4. No data was lost, but the partition is write-unavailable. The fix is more failure domains (racks/AZs), not just more replicas.
Open at this step: step 4, O3 rejected: NOT_ENOUGH_REPLICASWhat you start with
Independent-failure math makes RF=4 look bulletproof, but that assumption breaks when failures are correlated - a rack, power domain, or availability zone taking several replicas at once. Here four replicas span two racks with min.ISR=3; one rack fails as a unit, dropping the leader and a follower simultaneously.
Step by step
- Healthy cluster (4 replicas, 2 racks) t+1.000s RF=4, min.ISR=3. broker-1,2 sit in rack-a; broker-3,4 in rack-b. broker-1 leads. Open at this step: step 1, Healthy cluster (4 replicas, 2 racks)
- O1,O2 committed, HW=2 t+5.000s O1,O2 commit on all four replicas (HW=2). Open at this step: step 2, O1,O2 committed, HW=2
- rack-a down → ISR 2 (below floor) t+9.000s A whole failure domain - rack-a - goes down at once, taking the leader broker-1 AND broker-2 together. The survivors {broker-3, broker-4} in rack-b elect broker-3 (epoch→1) and still hold every committed record, but the ISR is now 2, below min.ISR=3. Open at this step: step 3, rack-a down → ISR 2 (below floor)
- O3 rejected: NOT_ENOUGH_REPLICAS t+12.000s O3 is refused with NOT_ENOUGH_REPLICAS - a single correlated rack failure breached the durability floor even at RF=4. No data was lost, but the partition is write-unavailable. The fix is more failure domains (racks/AZs), not just more replicas. Open at this step: step 4, O3 rejected: NOT_ENOUGH_REPLICAS
The takeaway
The survivors keep every committed record, but with the ISR below min.insync.replicas new writes are rejected - defending against correlated failure takes more failure domains (racks/AZs), not just a higher replication factor.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=08.0.6" title="Kafka Simulator: Correlated failure: a whole rack goes down" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/correlated-rack-failure/">Kafka rack failure: why RF=4 on two racks refuses writes</a></p>