Scenario 03.2.1 · Canonical failure

Losing two Kafka brokers with min.insync.replicas=2

Replication · Intermediate · about 2 min · Apache Kafka 4.3

Kill broker-2, then broker-3 under acks=all + min.ISR=2.

What you start with

The canonical durability setup: a 3-replica partition with acks=all and min.insync.replicas=2. Brokers are killed one after another while a producer writes, walking the partition from healthy, to degraded-but-writable, to below min.ISR.

Step by step

1. Cluster ready t+0.000s Open at this step: step 1, Cluster ready
2. Produce committed t+4.250s Open at this step: step 2, Produce committed
3. broker-2 hard kill t+8.500s broker-2 is offline. ISR shrinks on every partition it replicated to. Open at this step: step 3, broker-2 hard kill
4. Leader re-elected t+9.000s Open at this step: step 4, Leader re-elected
5. broker-3 hard kill t+14.800s Below min.insync.replicas - ISR is down to 1. Any acks=all write would now be refused with NotEnoughReplicas until a replica returns. Open at this step: step 5, broker-3 hard kill
6. Recovery (broker-2 restart) t+22.000s broker-2 rejoined; ISR is back to 2 on every partition. broker-3 is still down. Open at this step: step 6, Recovery (broker-2 restart)

The takeaway

As long as the ISR holds at least min.insync.replicas the partition keeps accepting acks=all writes; the second failure drops it below the floor and writes are refused until a replica returns.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.2.1" title="Kafka Simulator: Canonical failure" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/losing-two-brokers/">Losing two Kafka brokers with min.insync.replicas=2</a></p>

Further reading

Apache Kafka documentation

More in Replication

• Kafka ISR (in-sync replicas): how replicas leave and rejoin
• min.insync.replicas with acks=all: when Kafka refuses writes
• Kafka slow follower: replica.lag.time.max.ms and the ISR
• Kafka high watermark pinned by a slow in-sync replica
• Kafka preferred leader election after a rack failure
• Kafka unclean.leader.election.enable: offline or data loss
• Kafka unclean leader election: permanent loss of acked data
• Kafka log truncation when an old leader returns
• Kafka broker.rack: rack-aware replica placement
• Kafka rack-aware reassignment after adding a rack
• Increase the replication factor of an existing Kafka topic

All scenarios