Scenario 06.0.2 · KRaft quorum
Kafka KRaft quorum: losing a majority of controllers
A KRaft controller is a voter quorum: losing one of three voters keeps a majority (cluster fine), losing two leaves no majority - leader elections freeze - and restoring a single voter brings the majority back and the frozen election completes.
Run this scenario in the playgroundKRaft quorum - 3 voters, c0 active, c1/c2 standby. t+0.000s
The controller is a quorum, not a single node. Three voters (c0 active, c1/c2 standby) replicate the __cluster_metadata Raft log, and a metadata change commits once a majority of them has it. The quorum can keep serving as long as a strict MAJORITY of voters - 2 of 3 - is reachable.
Open at this step: step 1, KRaft quorum - 3 voters, c0 active, c1/c2 standbyStandby c2 killed - 2 of 3 voters, majority holds, c0 active. t+1.400s
One standby voter, c2, is killed. Two of three voters remain reachable - still a majority - so the quorum holds and c0 keeps serving as active controller. A KRaft cluster tolerates losing a minority of its voters with zero disruption to the control plane.
Open at this step: step 2, Standby c2 killed - 2 of 3 voters, majority holds, c0 activeLeader broker-1 killed - controller elects broker-2 (quorum intact). t+2.400s
Proof the control plane is healthy: kill the partition leader broker-1, and the active controller still runs the election, promoting broker-2 (leaderEpoch 1) and shrinking the ISR to the survivors. With a majority of voters up, leader elections complete normally.
Open at this step: step 3, Leader broker-1 killed - controller elects broker-2 (quorum intact)Fresh write commits - cluster fine with one voter down. t+3.000s
A fresh acks=all write also commits: the new leader broker-2 plus follower broker-3 satisfy min.ISR=2, so the high watermark advances to 2. Both the data plane and the control plane are fully functional with one of three voters down.
Open at this step: step 4, Fresh write commits - cluster fine with one voter downSecond voter c1 killed - only c0 left, no majority, quorum lost. t+3.400s
Now a SECOND voter, c1, is killed. Only c0 is left alive - 1 of 3 is NOT a majority - so the quorum is lost and there is no active controller (active is undefined). The metadata log can no longer commit changes. Existing leaders keep serving for the moment (broker-2 still leads), but the control plane is frozen.
Open at this step: step 5, Second voter c1 killed - only c0 left, no majority, quorum lostLeader broker-2 killed - election frozen, no quorum to elect. t+4.400s
With no quorum, kill the leader broker-2: the partition goes leaderless (leader=null, electing=true) but the election can never complete - there is no active controller to choose and publish a new leader. The partition stays stuck until a voter is restored (restart_controller) and a majority returns. This is why KRaft uses an odd voter count and needs a surviving majority.
Open at this step: step 6, Leader broker-2 killed - election frozen, no quorum to electVoter c1 restored - majority back, frozen election completes. t+5.000s
c1 is restarted. Two of three voters are reachable again - the majority is back - so c0 is the active controller once more. The frozen election completes immediately: broker-3, the last surviving ISR member (at LEO=HW), is elected leader of orders-0. Nothing was done to the brokers; restoring a single VOTER is what unblocked the data plane. The ISR is now one replica, below min.ISR=2, so acks=all writes wait for a broker to return - but the partition is led and readable again.
Open at this step: step 7, Voter c1 restored - majority back, frozen election completesWhat you start with
The KRaft controller is not a single node but a quorum of voters running the __cluster_metadata Raft log, and it can serve only while a strict majority is reachable (2 of 3 here). We kill one voter - majority holds, so a leader election still succeeds - then a second, so the next leader loss leaves its election frozen; then we restore one voter and watch the election complete.
Step by step
- KRaft quorum - 3 voters, c0 active, c1/c2 standby t+0.000s The controller is a quorum, not a single node. Three voters (c0 active, c1/c2 standby) replicate the __cluster_metadata Raft log, and a metadata change commits once a majority of them has it. The quorum can keep serving as long as a strict MAJORITY of voters - 2 of 3 - is reachable. Open at this step: step 1, KRaft quorum - 3 voters, c0 active, c1/c2 standby
- Standby c2 killed - 2 of 3 voters, majority holds, c0 active t+1.400s One standby voter, c2, is killed. Two of three voters remain reachable - still a majority - so the quorum holds and c0 keeps serving as active controller. A KRaft cluster tolerates losing a minority of its voters with zero disruption to the control plane. Open at this step: step 2, Standby c2 killed - 2 of 3 voters, majority holds, c0 active
- Leader broker-1 killed - controller elects broker-2 (quorum intact) t+2.400s Proof the control plane is healthy: kill the partition leader broker-1, and the active controller still runs the election, promoting broker-2 (leaderEpoch 1) and shrinking the ISR to the survivors. With a majority of voters up, leader elections complete normally. Open at this step: step 3, Leader broker-1 killed - controller elects broker-2 (quorum intact)
- Fresh write commits - cluster fine with one voter down t+3.000s A fresh acks=all write also commits: the new leader broker-2 plus follower broker-3 satisfy min.ISR=2, so the high watermark advances to 2. Both the data plane and the control plane are fully functional with one of three voters down. Open at this step: step 4, Fresh write commits - cluster fine with one voter down
- Second voter c1 killed - only c0 left, no majority, quorum lost t+3.400s Now a SECOND voter, c1, is killed. Only c0 is left alive - 1 of 3 is NOT a majority - so the quorum is lost and there is no active controller (active is undefined). The metadata log can no longer commit changes. Existing leaders keep serving for the moment (broker-2 still leads), but the control plane is frozen. Open at this step: step 5, Second voter c1 killed - only c0 left, no majority, quorum lost
- Leader broker-2 killed - election frozen, no quorum to elect t+4.400s With no quorum, kill the leader broker-2: the partition goes leaderless (leader=null, electing=true) but the election can never complete - there is no active controller to choose and publish a new leader. The partition stays stuck until a voter is restored (restart_controller) and a majority returns. This is why KRaft uses an odd voter count and needs a surviving majority. Open at this step: step 6, Leader broker-2 killed - election frozen, no quorum to elect
- Voter c1 restored - majority back, frozen election completes t+5.000s c1 is restarted. Two of three voters are reachable again - the majority is back - so c0 is the active controller once more. The frozen election completes immediately: broker-3, the last surviving ISR member (at LEO=HW), is elected leader of orders-0. Nothing was done to the brokers; restoring a single VOTER is what unblocked the data plane. The ISR is now one replica, below min.ISR=2, so acks=all writes wait for a broker to return - but the partition is led and readable again. Open at this step: step 7, Voter c1 restored - majority back, frozen election completes
The takeaway
Quorum availability, not broker count, is what keeps the control plane alive: with a majority gone there is no active controller and a leaderless partition stays that way - and it is restoring a voter, not a broker, that brings the election back.
Configs in this scenario
-
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works -
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=06.0.2" title="Kafka Simulator: KRaft quorum" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/kraft-quorum/">Kafka KRaft quorum: losing a majority of controllers</a></p>