Scenario 06.0.1 · The controller's job
What the Kafka controller does when a broker dies
The active controller watches broker liveness: a dead follower only shrinks the ISR in metadata, a dead leader makes it elect a new one from the surviving ISR - and neither touches the controller epoch.
Run this scenario in the playgroundKRaft cluster ready - c0 active, broker-1 leads orders-0. t+0.000s
Every cluster elects exactly one active controller. Here it is a KRaft quorum of three voters with c0 active; c1 and c2 are hot standbys replicating its metadata log. The controller owns cluster metadata - including which broker leads each partition and which replicas are in sync. broker-1 leads orders-0, with broker-2 and broker-3 as in-sync followers.
Open at this step: step 1, KRaft cluster ready - c0 active, broker-1 leads orders-0Two records committed - broker-1 leads, HW 2, ISR full. t+0.800s
Producer p1 (acks=all) appends two records to the leader broker-1; both followers replicate them, so the high watermark advances to 2 and the ISR stays full at three. This is a pure data-plane path between brokers - the controller is not involved in producing or replicating, only in deciding who leads and who counts as in sync.
Open at this step: step 2, Two records committed - broker-1 leads, HW 2, ISR fullFollower broker-3 killed - ISR shrinks, no election. t+1.200s
broker-3 is hard-killed. It was only a follower, so broker-1 keeps leading and the partition keeps serving. The controller's job here is bookkeeping: it notices the dead broker, drops it from the ISR and republishes the partition metadata. Nothing is elected - leaderEpoch stays 0 - and the controller epoch stays 0 too. Most of a controller's life looks like this frame.
Open at this step: step 3, Follower broker-3 killed - ISR shrinks, no electionLeader broker-1 killed - partition leaderless, electing. t+1.900s
Now broker-1 is hard-killed. It led orders-0, so the partition is suddenly leaderless: leader=null and electing=true, and it accepts no reads or writes. This is the case the controller exists for: c0 detects the dead leader (its heartbeat stops) and begins a leader election.
Open at this step: step 4, Leader broker-1 killed - partition leaderless, electingController elected broker-2 - leaderEpoch 1, controller epoch still 0. t+2.500s
c0 elects broker-2 - the only surviving ISR member, at LEO=HW - as the new leader and publishes it in metadata. The partition's leaderEpoch bumps 0→1 (fencing the stale broker-1); the high watermark is unchanged, so no data was lost. Note the CONTROLLER epoch is still 0 after both failures: data brokers died, not the controller, so the control plane never changed terms.
Open at this step: step 5, Controller elected broker-2 - leaderEpoch 1, controller epoch still 0What you start with
Every Kafka cluster has exactly one active controller - here a KRaft quorum of three voters. Its job is to watch broker liveness and keep partition metadata true, and most of the time that means no election at all. We give orders-0 committed data, then kill a follower (the ISR shrinks, broker-1 still leads) and only then its leader - now the controller must elect, and it promotes broker-2.
Step by step
- KRaft cluster ready - c0 active, broker-1 leads orders-0 t+0.000s Every cluster elects exactly one active controller. Here it is a KRaft quorum of three voters with c0 active; c1 and c2 are hot standbys replicating its metadata log. The controller owns cluster metadata - including which broker leads each partition and which replicas are in sync. broker-1 leads orders-0, with broker-2 and broker-3 as in-sync followers. Open at this step: step 1, KRaft cluster ready - c0 active, broker-1 leads orders-0
- Two records committed - broker-1 leads, HW 2, ISR full t+0.800s Producer p1 (acks=all) appends two records to the leader broker-1; both followers replicate them, so the high watermark advances to 2 and the ISR stays full at three. This is a pure data-plane path between brokers - the controller is not involved in producing or replicating, only in deciding who leads and who counts as in sync. Open at this step: step 2, Two records committed - broker-1 leads, HW 2, ISR full
- Follower broker-3 killed - ISR shrinks, no election t+1.200s broker-3 is hard-killed. It was only a follower, so broker-1 keeps leading and the partition keeps serving. The controller's job here is bookkeeping: it notices the dead broker, drops it from the ISR and republishes the partition metadata. Nothing is elected - leaderEpoch stays 0 - and the controller epoch stays 0 too. Most of a controller's life looks like this frame. Open at this step: step 3, Follower broker-3 killed - ISR shrinks, no election
- Leader broker-1 killed - partition leaderless, electing t+1.900s Now broker-1 is hard-killed. It led orders-0, so the partition is suddenly leaderless: leader=null and electing=true, and it accepts no reads or writes. This is the case the controller exists for: c0 detects the dead leader (its heartbeat stops) and begins a leader election. Open at this step: step 4, Leader broker-1 killed - partition leaderless, electing
- Controller elected broker-2 - leaderEpoch 1, controller epoch still 0 t+2.500s c0 elects broker-2 - the only surviving ISR member, at LEO=HW - as the new leader and publishes it in metadata. The partition's leaderEpoch bumps 0→1 (fencing the stale broker-1); the high watermark is unchanged, so no data was lost. Note the CONTROLLER epoch is still 0 after both failures: data brokers died, not the controller, so the control plane never changed terms. Open at this step: step 5, Controller elected broker-2 - leaderEpoch 1, controller epoch still 0
The takeaway
A dead follower is a metadata update (the ISR shrinks, leaderEpoch stays); a dead leader is an election (leaderEpoch bumps). Both leave the controller epoch alone - the control plane only changes terms when the controller itself changes.
Configs in this scenario
-
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=06.0.1" title="Kafka Simulator: The controller's job" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/controllers-job/">What the Kafka controller does when a broker dies</a></p>