Scenario 08.0.4 · Cascading failure: losing the ISR one by one

Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one

Chaos · Intermediate · about 4 min · Apache Kafka 4.3

Brokers fall in sequence; the ISR shrinks step by step until it drops below min.ISR - then acks=all stops committing (NOT_ENOUGH_REPLICAS) and the high watermark freezes.

What you start with

Correlated outages often unfold in sequence rather than all at once. This RF=5 / min.ISR=3 partition loses brokers one at a time, and you watch the ISR shrink toward the durability floor - each write still commits while the ISR meets min.ISR, until one more death drops it below.

Step by step

1. Healthy cluster (ISR 5) t+1.000s RF=5, min.ISR=3. broker-1 leads; all five replicas are in-sync. Open at this step: step 1, Healthy cluster (ISR 5)
2. O1 committed t+3.000s O1 commits on all five (HW=1). Open at this step: step 2, O1 committed
3. broker-5 down → ISR 4 t+5.000s broker-5 is killed; the ISR shrinks to 4 - still well above the min.ISR floor of 3. Open at this step: step 3, broker-5 down → ISR 4
4. O2 still committed t+7.000s O2 commits on the ISR of 4 (≥3) and the HW advances to 2. Writes are unaffected. Open at this step: step 4, O2 still committed
5. broker-4 down → ISR 3 (floor) t+9.000s broker-4 is killed; the ISR is now {broker-1, broker-2, broker-3} = exactly min.ISR. The next write is the last one that can commit. Open at this step: step 5, broker-4 down → ISR 3 (floor)
6. O3 committed at floor t+11.000s O3 commits on the ISR of exactly 3 - HW=3. acks=all is satisfied because ISR == min.ISR. Open at this step: step 6, O3 committed at floor
7. broker-3 down → ISR 2 (below floor) t+13.000s broker-3 is killed; the ISR drops to {broker-1, broker-2} = 2, below min.ISR=3. The leader is still alive, but durability can no longer be guaranteed. Open at this step: step 7, broker-3 down → ISR 2 (below floor)
8. O4 rejected: NOT_ENOUGH_REPLICAS t+15.000s O4 is refused with NOT_ENOUGH_REPLICAS and the producer is left retrying; the HW freezes at 3. Availability under acks=all is bounded by min.ISR, not by whether the leader is up. Open at this step: step 8, O4 rejected: NOT_ENOUGH_REPLICAS

The takeaway

Under acks=all, availability is bounded by min.insync.replicas, not by whether the leader is up: once the ISR falls below the floor, writes are rejected with NOT_ENOUGH_REPLICAS and the high watermark freezes even though the cluster is otherwise healthy.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=08.0.4" title="Kafka Simulator: Cascading failure: losing the ISR one by one" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/cascading-isr-loss/">Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• NotEnoughReplicasException in Kafka: causes and fixes
• Kafka under-replicated partitions: causes and fixes

More in Chaos

• Kafka broker failure: how the partition leader fails over
• Kafka split-brain: what a network partition does to leaders
• Kafka network partition: the majority side keeps writing
• Kafka ISR re-expansion: how a restarted broker catches up
• Kafka rack failure: why RF=4 on two racks refuses writes

All scenarios