Scenario 06.0.3 · Controller failover & metadata catch-up
Kafka KRaft controller failover and metadata log catch-up
A controller keeps cluster state in the replicated __cluster_metadata log. A restarted voter must replay the records it missed; on failover the new active controller needs a short start-up before it serves, and partition leader elections wait until it does.
Run this scenario in the playgroundQuorum healthy. t+0.350s
Three KRaft voters run the __cluster_metadata log, all at committed offset 0. c0 is the active controller - the single writer - with c1 and c2 as standbys. One record (O1) is committed to orders-0 (HW=1).
Open at this step: step 1, Quorum healthyc0 appends a metadata record. t+0.900s
The active controller drives a metadata change (a retention.ms edit), appending one record to __cluster_metadata: c0's offset advances 0→1. The standbys are still at 0 - they replicate on a cadence, not instantly. The simulator counts the append as committed now; in KRaft it commits once a majority of voters (2 of 3) has it.
Open at this step: step 2, c0 appends a metadata recordStandbys replicate. t+1.100s
The metadata_replicate loop fires: standbys c1 and c2 pull the new record and advance to offset 1. The quorum is level again - every voter holds the same committed log.
Open at this step: step 3, Standbys replicateKill standby c2. t+1.300s
Standby c2 is killed. It stops voting and its log freezes at offset 1. Two of three voters remain reachable - a majority - so c0 keeps serving.
Open at this step: step 4, Kill standby c2Log advances while c2 is down. t+1.900s
While c2 is down, c0 drives two more metadata changes: its offset climbs to 3. Dead c2 misses both records and stays at 1; live standby c1 hasn't replicated the burst yet. With c2 down, KRaft commits these records only once c1 has them too.
Open at this step: step 5, Log advances while c2 is downLive standby stays caught up. t+2.100s
On the next replication tick, live standby c1 catches up to offset 3. Dead c2 cannot replicate - it is still stuck at offset 1, now two records behind.
Open at this step: step 6, Live standby stays caught upRestart c2 - behind. t+2.600s
c2 is restarted. It rejoins the quorum but returns BEHIND - its log is at offset 1 while the committed offset is 3. A returning voter is not instantly useful; it must replay the records it missed.
Open at this step: step 7, Restart c2 - behindc2 replays the log. t+3.100s
c2 replays __cluster_metadata via the replication loop and reaches offset 3. It is fully caught up and back in sync with the rest of the quorum.
Open at this step: step 8, c2 replays the logc0 appends offset 4. t+3.500s
c0 drives one more change and appends offset 4. The simulator counts it committed at once, but in KRaft it is not: only c0 has it, and a record commits once a majority (2 of 3 voters) does. The standbys lag at 3 until the next replication tick.
Open at this step: step 9, c0 appends offset 4Active c0 dies; c1 elected. t+3.700s
The active controller c0 is killed before the standbys replicated offset 4. c1 is elected and the controller EPOCH bumps 0→1 (a new term). The simulator shows c1 at 3, behind c0's 4, and holds it in a catch-up window before it serves. In KRaft c1 already has every committed record: offset 4 was only on c0, so it was never committed, and c0 truncates it when it rejoins.
Open at this step: step 10, Active c0 dies; c1 electedElection FROZEN during catch-up. t+4.700s
The partition leader (broker-1) is killed during the catch-up window. An election is needed, but no controller is serving yet - c1 is still in the simulator's catch-up window - so the election FREEZES: the partition is leaderless (electing, no leader). The simulator also copies offset 4 to standby c2 here, though only the dead c0 held it; in KRaft no voter can fetch that record, and this window is just the short start-up before c1 commits a record in its new epoch.
Open at this step: step 11, Election FROZEN during catch-upc1 catches up → election completes. t+5.300s
c1 reaches offset 4 in the simulator and starts serving. The frozen election immediately resumes and broker-2, an in-sync survivor, is elected the new leader. The lesson holds in real KRaft: partition elections wait while no controller is serving, and a new active controller serves once it has committed a record in its new epoch.
Open at this step: step 12, c1 catches up → election completesWhat you start with
The controller quorum stores cluster state in the replicated __cluster_metadata log: the active controller is the sole writer and standbys replicate its records on a cadence. This scenario shows a standby restarted after the log advanced, which replays what it missed, and a failover, where partition elections wait until the new active controller serves. To make that window visible the simulator counts the leader's append as committed at once and shows the new controller catching up; real KRaft commits a record only once a majority of voters has it, and elects only a voter that holds every committed record.
Step by step
- Quorum healthy t+0.350s Three KRaft voters run the __cluster_metadata log, all at committed offset 0. c0 is the active controller - the single writer - with c1 and c2 as standbys. One record (O1) is committed to orders-0 (HW=1). Open at this step: step 1, Quorum healthy
- c0 appends a metadata record t+0.900s The active controller drives a metadata change (a retention.ms edit), appending one record to __cluster_metadata: c0's offset advances 0→1. The standbys are still at 0 - they replicate on a cadence, not instantly. The simulator counts the append as committed now; in KRaft it commits once a majority of voters (2 of 3) has it. Open at this step: step 2, c0 appends a metadata record
- Standbys replicate t+1.100s The metadata_replicate loop fires: standbys c1 and c2 pull the new record and advance to offset 1. The quorum is level again - every voter holds the same committed log. Open at this step: step 3, Standbys replicate
- Kill standby c2 t+1.300s Standby c2 is killed. It stops voting and its log freezes at offset 1. Two of three voters remain reachable - a majority - so c0 keeps serving. Open at this step: step 4, Kill standby c2
- Log advances while c2 is down t+1.900s While c2 is down, c0 drives two more metadata changes: its offset climbs to 3. Dead c2 misses both records and stays at 1; live standby c1 hasn't replicated the burst yet. With c2 down, KRaft commits these records only once c1 has them too. Open at this step: step 5, Log advances while c2 is down
- Live standby stays caught up t+2.100s On the next replication tick, live standby c1 catches up to offset 3. Dead c2 cannot replicate - it is still stuck at offset 1, now two records behind. Open at this step: step 6, Live standby stays caught up
- Restart c2 - behind t+2.600s c2 is restarted. It rejoins the quorum but returns BEHIND - its log is at offset 1 while the committed offset is 3. A returning voter is not instantly useful; it must replay the records it missed. Open at this step: step 7, Restart c2 - behind
- c2 replays the log t+3.100s c2 replays __cluster_metadata via the replication loop and reaches offset 3. It is fully caught up and back in sync with the rest of the quorum. Open at this step: step 8, c2 replays the log
- c0 appends offset 4 t+3.500s c0 drives one more change and appends offset 4. The simulator counts it committed at once, but in KRaft it is not: only c0 has it, and a record commits once a majority (2 of 3 voters) does. The standbys lag at 3 until the next replication tick. Open at this step: step 9, c0 appends offset 4
- Active c0 dies; c1 elected t+3.700s The active controller c0 is killed before the standbys replicated offset 4. c1 is elected and the controller EPOCH bumps 0→1 (a new term). The simulator shows c1 at 3, behind c0's 4, and holds it in a catch-up window before it serves. In KRaft c1 already has every committed record: offset 4 was only on c0, so it was never committed, and c0 truncates it when it rejoins. Open at this step: step 10, Active c0 dies; c1 elected
- Election FROZEN during catch-up t+4.700s The partition leader (broker-1) is killed during the catch-up window. An election is needed, but no controller is serving yet - c1 is still in the simulator's catch-up window - so the election FREEZES: the partition is leaderless (electing, no leader). The simulator also copies offset 4 to standby c2 here, though only the dead c0 held it; in KRaft no voter can fetch that record, and this window is just the short start-up before c1 commits a record in its new epoch. Open at this step: step 11, Election FROZEN during catch-up
- c1 catches up → election completes t+5.300s c1 reaches offset 4 in the simulator and starts serving. The frozen election immediately resumes and broker-2, an in-sync survivor, is elected the new leader. The lesson holds in real KRaft: partition elections wait while no controller is serving, and a new active controller serves once it has committed a record in its new epoch. Open at this step: step 12, c1 catches up → election completes
The takeaway
A restarted voter replays the records it missed before it is back in sync, and a partition leader killed while no controller is serving has its election frozen until the new active controller takes over. In real KRaft that start-up is short: the new controller already holds every committed record, a record only the old leader had was never committed and is truncated when that voter rejoins, and the new controller serves once it commits a record in its new epoch.
Configs in this scenario
-
retention.How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms worksms
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=06.0.3" title="Kafka Simulator: Controller failover & metadata catch-up" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/controller-failover-catchup/">Kafka KRaft controller failover and metadata log catch-up</a></p>