---
title: "Kafka broker failure: how the partition leader fails over"
description: "Hard-kill the leader of an RF=3, min.insync.replicas=2 Kafka partition. The ISR shrinks to two, broker-2 takes over and both committed records survive."
url: https://monedula.dev/kafka-simulator/scenarios/single-broker-hard-kill/
---

# Kafka broker failure: how the partition leader fails over

Scenario 08.0.1 · Single broker hard kill. Chaos · Intermediate · about 3 min · Apache Kafka 4.3.

Hard-kill the partition leader: the ISR shrinks to the survivors, a new leader is elected, and every committed record stays durable - writes continue at the min.ISR floor.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1

## What you start with

A hard kill is an abrupt broker death - power loss, OOM, kernel panic, no graceful drain. With acks=all, min.ISR=2 and RF=3, two records commit on all three replicas; then the partition leader (broker-1) is killed and you watch the controller shrink the ISR to the two survivors and elect a new leader from the in-sync set.

## Step by step

1. **Healthy cluster** (t+1.000s): Three brokers, RF=3, min.ISR=2. broker-1 leads partition 0; broker-2 and broker-3 are in-sync followers. Nothing produced yet (HW=0). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1&t=0)
2. **O1 committed** (t+3.000s): O1 is appended on the leader, replicated to both followers, and the high watermark advances to 1 - the record is durable on all three replicas. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1&t=1)
3. **O2 committed** (t+5.000s): O2 commits the same way. HW=2: both records are durable on broker-1, broker-2 and broker-3. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1&t=2)
4. **Leader killed** (t+8.000s): broker-1 (the leader) is hard-killed. The controller drops it from the ISR (now {broker-2, broker-3}) and elects broker-2 as the new leader (leaderEpoch 0→1). HW stays 2 - both committed records survived on the followers, so the kill cost no data. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1&t=3)
5. **Still writable** (t+11.000s): The surviving ISR of 2 still meets min.ISR, so the new leader accepts O3 and the HW advances to 3. A single broker loss is a non-event for a RF=3 / min.ISR=2 topic. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.1&t=4)

## The takeaway

Losing a single broker is a non-event for durability: every committed record survives on the in-sync replicas and writes keep flowing, as long as the survivors still satisfy min.insync.replicas.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#replication)