---
title: "What the Kafka controller does when a broker dies"
description: "A dead follower only shrinks the ISR; a dead leader makes the KRaft controller elect a new one and bump leaderEpoch. The Kafka controller epoch never moves."
url: https://monedula.dev/kafka-simulator/scenarios/controllers-job/
---

# What the Kafka controller does when a broker dies

Scenario 06.0.1 · The controller's job. KRaft · Intermediate · about 3 min · Apache Kafka 4.3.

The active controller watches broker liveness: a dead follower only shrinks the ISR in metadata, a dead leader makes it elect a new one from the surviving ISR - and neither touches the controller epoch.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1

## What you start with

Every Kafka cluster has exactly one active controller - here a KRaft quorum of three voters. Its job is to watch broker liveness and keep partition metadata true, and most of the time that means no election at all. We give orders-0 committed data, then kill a follower (the ISR shrinks, broker-1 still leads) and only then its leader - now the controller must elect, and it promotes broker-2.

## Step by step

1. **KRaft cluster ready - c0 active, broker-1 leads orders-0** (t+0.000s): Every cluster elects exactly one active controller. Here it is a KRaft quorum of three voters with c0 active; c1 and c2 are hot standbys replicating its metadata log. The controller owns cluster metadata - including which broker leads each partition and which replicas are in sync. broker-1 leads orders-0, with broker-2 and broker-3 as in-sync followers. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1&t=0)
2. **Two records committed - broker-1 leads, HW 2, ISR full** (t+0.800s): Producer p1 (acks=all) appends two records to the leader broker-1; both followers replicate them, so the high watermark advances to 2 and the ISR stays full at three. This is a pure data-plane path between brokers - the controller is not involved in producing or replicating, only in deciding who leads and who counts as in sync. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1&t=1)
3. **Follower broker-3 killed - ISR shrinks, no election** (t+1.200s): broker-3 is hard-killed. It was only a follower, so broker-1 keeps leading and the partition keeps serving. The controller's job here is bookkeeping: it notices the dead broker, drops it from the ISR and republishes the partition metadata. Nothing is elected - leaderEpoch stays 0 - and the controller epoch stays 0 too. Most of a controller's life looks like this frame. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1&t=2)
4. **Leader broker-1 killed - partition leaderless, electing** (t+1.900s): Now broker-1 is hard-killed. It led orders-0, so the partition is suddenly leaderless: leader=null and electing=true, and it accepts no reads or writes. This is the case the controller exists for: c0 detects the dead leader (its heartbeat stops) and begins a leader election. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1&t=3)
5. **Controller elected broker-2 - leaderEpoch 1, controller epoch still 0** (t+2.500s): c0 elects broker-2 - the only surviving ISR member, at LEO=HW - as the new leader and publishes it in metadata. The partition's leaderEpoch bumps 0→1 (fencing the stale broker-1); the high watermark is unchanged, so no data was lost. Note the CONTROLLER epoch is still 0 after both failures: data brokers died, not the controller, so the control plane never changed terms. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.1&t=4)

## The takeaway

A dead follower is a metadata update (the ISR shrinks, leaderEpoch stays); a dead leader is an election (leaderEpoch bumps). Both leave the controller epoch alone - the control plane only changes terms when the controller itself changes.

## Configs in this scenario

- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#replication)