Scenario 08.0.1 · Single broker hard kill
Kafka broker failure: how the partition leader fails over
Hard-kill the partition leader: the ISR shrinks to the survivors, a new leader is elected, and every committed record stays durable - writes continue at the min.ISR floor.
Run this scenario in the playgroundHealthy cluster. t+1.000s
Three brokers, RF=3, min.ISR=2. broker-1 leads partition 0; broker-2 and broker-3 are in-sync followers. Nothing produced yet (HW=0).
Open at this step: step 1, Healthy clusterO1 committed. t+3.000s
O1 is appended on the leader, replicated to both followers, and the high watermark advances to 1 - the record is durable on all three replicas.
Open at this step: step 2, O1 committedO2 committed. t+5.000s
O2 commits the same way. HW=2: both records are durable on broker-1, broker-2 and broker-3.
Open at this step: step 3, O2 committedLeader killed. t+8.000s
broker-1 (the leader) is hard-killed. The controller drops it from the ISR (now {broker-2, broker-3}) and elects broker-2 as the new leader (leaderEpoch 0→1). HW stays 2 - both committed records survived on the followers, so the kill cost no data.
Open at this step: step 4, Leader killedStill writable. t+11.000s
The surviving ISR of 2 still meets min.ISR, so the new leader accepts O3 and the HW advances to 3. A single broker loss is a non-event for a RF=3 / min.ISR=2 topic.
Open at this step: step 5, Still writableWhat you start with
A hard kill is an abrupt broker death - power loss, OOM, kernel panic, no graceful drain. With acks=all, min.ISR=2 and RF=3, two records commit on all three replicas; then the partition leader (broker-1) is killed and you watch the controller shrink the ISR to the two survivors and elect a new leader from the in-sync set.
Step by step
- Healthy cluster t+1.000s Three brokers, RF=3, min.ISR=2. broker-1 leads partition 0; broker-2 and broker-3 are in-sync followers. Nothing produced yet (HW=0). Open at this step: step 1, Healthy cluster
- O1 committed t+3.000s O1 is appended on the leader, replicated to both followers, and the high watermark advances to 1 - the record is durable on all three replicas. Open at this step: step 2, O1 committed
- O2 committed t+5.000s O2 commits the same way. HW=2: both records are durable on broker-1, broker-2 and broker-3. Open at this step: step 3, O2 committed
- Leader killed t+8.000s broker-1 (the leader) is hard-killed. The controller drops it from the ISR (now {broker-2, broker-3}) and elects broker-2 as the new leader (leaderEpoch 0→1). HW stays 2 - both committed records survived on the followers, so the kill cost no data. Open at this step: step 4, Leader killed
- Still writable t+11.000s The surviving ISR of 2 still meets min.ISR, so the new leader accepts O3 and the HW advances to 3. A single broker loss is a non-event for a RF=3 / min.ISR=2 topic. Open at this step: step 5, Still writable
The takeaway
Losing a single broker is a non-event for durability: every committed record survives on the in-sync replicas and writes keep flowing, as long as the survivors still satisfy min.insync.replicas.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=08.0.1" title="Kafka Simulator: Single broker hard kill" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/single-broker-hard-kill/">Kafka broker failure: how the partition leader fails over</a></p>