--- title: "Kafka KRaft controller failover and metadata log catch-up" description: "A restarted KRaft voter replays the __cluster_metadata records it missed, and on failover a Kafka partition election waits until the new controller serves." url: https://monedula.dev/kafka-simulator/scenarios/controller-failover-catchup/ --- # Kafka KRaft controller failover and metadata log catch-up Scenario 06.0.3 · Controller failover & metadata catch-up. KRaft · Advanced · about 4 min · Apache Kafka 4.3. A controller keeps cluster state in the replicated __cluster_metadata log. A restarted voter must replay the records it missed; on failover the new active controller needs a short start-up before it serves, and partition leader elections wait until it does. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3 ## What you start with The controller quorum stores cluster state in the replicated __cluster_metadata log: the active controller is the sole writer and standbys replicate its records on a cadence. This scenario shows a standby restarted after the log advanced, which replays what it missed, and a failover, where partition elections wait until the new active controller serves. To make that window visible the simulator counts the leader's append as committed at once and shows the new controller catching up; real KRaft commits a record only once a majority of voters has it, and elects only a voter that holds every committed record. ## Step by step 1. **Quorum healthy** (t+0.350s): Three KRaft voters run the __cluster_metadata log, all at committed offset 0. c0 is the active controller - the single writer - with c1 and c2 as standbys. One record (O1) is committed to orders-0 (HW=1). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=0) 2. **c0 appends a metadata record** (t+0.900s): The active controller drives a metadata change (a retention.ms edit), appending one record to __cluster_metadata: c0's offset advances 0→1. The standbys are still at 0 - they replicate on a cadence, not instantly. The simulator counts the append as committed now; in KRaft it commits once a majority of voters (2 of 3) has it. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=1) 3. **Standbys replicate** (t+1.100s): The metadata_replicate loop fires: standbys c1 and c2 pull the new record and advance to offset 1. The quorum is level again - every voter holds the same committed log. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=2) 4. **Kill standby c2** (t+1.300s): Standby c2 is killed. It stops voting and its log freezes at offset 1. Two of three voters remain reachable - a majority - so c0 keeps serving. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=3) 5. **Log advances while c2 is down** (t+1.900s): While c2 is down, c0 drives two more metadata changes: its offset climbs to 3. Dead c2 misses both records and stays at 1; live standby c1 hasn't replicated the burst yet. With c2 down, KRaft commits these records only once c1 has them too. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=4) 6. **Live standby stays caught up** (t+2.100s): On the next replication tick, live standby c1 catches up to offset 3. Dead c2 cannot replicate - it is still stuck at offset 1, now two records behind. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=5) 7. **Restart c2 - behind** (t+2.600s): c2 is restarted. It rejoins the quorum but returns BEHIND - its log is at offset 1 while the committed offset is 3. A returning voter is not instantly useful; it must replay the records it missed. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=6) 8. **c2 replays the log** (t+3.100s): c2 replays __cluster_metadata via the replication loop and reaches offset 3. It is fully caught up and back in sync with the rest of the quorum. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=7) 9. **c0 appends offset 4** (t+3.500s): c0 drives one more change and appends offset 4. The simulator counts it committed at once, but in KRaft it is not: only c0 has it, and a record commits once a majority (2 of 3 voters) does. The standbys lag at 3 until the next replication tick. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=8) 10. **Active c0 dies; c1 elected** (t+3.700s): The active controller c0 is killed before the standbys replicated offset 4. c1 is elected and the controller EPOCH bumps 0→1 (a new term). The simulator shows c1 at 3, behind c0's 4, and holds it in a catch-up window before it serves. In KRaft c1 already has every committed record: offset 4 was only on c0, so it was never committed, and c0 truncates it when it rejoins. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=9) 11. **Election FROZEN during catch-up** (t+4.700s): The partition leader (broker-1) is killed during the catch-up window. An election is needed, but no controller is serving yet - c1 is still in the simulator's catch-up window - so the election FREEZES: the partition is leaderless (electing, no leader). The simulator also copies offset 4 to standby c2 here, though only the dead c0 held it; in KRaft no voter can fetch that record, and this window is just the short start-up before c1 commits a record in its new epoch. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=10) 12. **c1 catches up → election completes** (t+5.300s): c1 reaches offset 4 in the simulator and starts serving. The frozen election immediately resumes and broker-2, an in-sync survivor, is elected the new leader. The lesson holds in real KRaft: partition elections wait while no controller is serving, and a new active controller serves once it has committed a record in its new epoch. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=06.0.3&t=11) ## The takeaway A restarted voter replays the records it missed before it is back in sync, and a partition leader killed while no controller is serving has its election frozen until the new active controller takes over. In real KRaft that start-up is short: the new controller already holds every committed record, a record only the old leader had was never committed and is truncated when that voter rejoins, and the new controller serves once it commits a record in its new epoch. ## Configs in this scenario - [`retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms): How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. [How retention.ms works](https://monedula.dev/kafka-simulator/configs/retention-ms/) ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/operations/kraft/)