---
title: "Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one"
description: "Kill the replicas of an RF=5, min.insync.replicas=3 Kafka partition one by one. Writes commit down to an ISR of 3, then O4 is rejected and the HW freezes."
url: https://monedula.dev/kafka-simulator/scenarios/cascading-isr-loss/
---

# Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one

Scenario 08.0.4 · Cascading failure: losing the ISR one by one. Chaos · Intermediate · about 4 min · Apache Kafka 4.3.

Brokers fall in sequence; the ISR shrinks step by step until it drops below min.ISR - then acks=all stops committing (NOT_ENOUGH_REPLICAS) and the high watermark freezes.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4

## What you start with

Correlated outages often unfold in sequence rather than all at once. This RF=5 / min.ISR=3 partition loses brokers one at a time, and you watch the ISR shrink toward the durability floor - each write still commits while the ISR meets min.ISR, until one more death drops it below.

## Step by step

1. **Healthy cluster (ISR 5)** (t+1.000s): RF=5, min.ISR=3. broker-1 leads; all five replicas are in-sync. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=0)
2. **O1 committed** (t+3.000s): O1 commits on all five (HW=1). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=1)
3. **broker-5 down → ISR 4** (t+5.000s): broker-5 is killed; the ISR shrinks to 4 - still well above the min.ISR floor of 3. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=2)
4. **O2 still committed** (t+7.000s): O2 commits on the ISR of 4 (≥3) and the HW advances to 2. Writes are unaffected. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=3)
5. **broker-4 down → ISR 3 (floor)** (t+9.000s): broker-4 is killed; the ISR is now {broker-1, broker-2, broker-3} = exactly min.ISR. The next write is the last one that can commit. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=4)
6. **O3 committed at floor** (t+11.000s): O3 commits on the ISR of exactly 3 - HW=3. acks=all is satisfied because ISR == min.ISR. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=5)
7. **broker-3 down → ISR 2 (below floor)** (t+13.000s): broker-3 is killed; the ISR drops to {broker-1, broker-2} = 2, below min.ISR=3. The leader is still alive, but durability can no longer be guaranteed. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=6)
8. **O4 rejected: NOT_ENOUGH_REPLICAS** (t+15.000s): O4 is refused with NOT_ENOUGH_REPLICAS and the producer is left retrying; the HW freezes at 3. Availability under acks=all is bounded by min.ISR, not by whether the leader is up. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.4&t=7)

## The takeaway

Under acks=all, availability is bounded by min.insync.replicas, not by whether the leader is up: once the ISR falls below the floor, writes are rejected with NOT_ENOUGH_REPLICAS and the high watermark freezes even though the cluster is otherwise healthy.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas)

## Troubleshooting

- [NotEnoughReplicasException in Kafka: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/not-enough-replicas/)
- [Kafka under-replicated partitions: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/under-replicated-partitions/)