Scenario 09.1.5 · Losing a whole DC

Kafka data center failure: leader failover to the other DC

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3

kill_dc takes out the leader DC; the witness keeps the voter quorum, so leadership fails over to the surviving DC and writes resume.

Run this scenario in the playground

O3 accepted on the new DC-B leader - cluster stayed available. t+12.050s

A fresh acks=all write commits on the failed-over DC-B leader (ISR 2 = min.ISR) with O1/O2 intact. The witness is what made this possible: without it an even 2-voter split would have lost quorum and the election would have frozen (see 09.1.4).

Open at this step: step 5, O3 accepted on the new DC-B leader - cluster stayed available

What you start with

Same stretched two-DC topology as 09.1.1, but here we kill the DC that holds the partition leader and one of the controller voters. A witness voter in a third site keeps a 2-of-3 controller majority, and the surviving DC’s in-sync replicas let the controller elect a new leader there. A fresh write then commits on the new leader.

Step by step

Stretched 2-DC cluster ready - leader in DC-A t+0.000s The same stretched topology as 09.1.1: RF 4 across DC-A + DC-B, min.ISR 2, a voter in each data DC plus a dc-c witness. The partition leader is one of the DC-A brokers. Open at this step: step 1, Stretched 2-DC cluster ready - leader in DC-A
O1 committed - durable across both DCs t+3.050s p1 produces with acks=all; O1 commits only after the cross-DC ISR has it, so it is already replicated into DC-B before the producer is acked. Open at this step: step 2, O1 committed - durable across both DCs
O2 committed - log {O1, O2} on all 4 replicas t+6.050s A second acks=all write, fully replicated across both data centres. Open at this step: step 3, O2 committed - log {O1, O2} on all 4 replicas
DC-A killed - quorum holds, leadership fails over to DC-B t+9.600s kill_dc takes out both DC-A brokers (leader + follower) AND the DC-A voter. c-b plus the dc-c witness are still a 2-of-3 majority, so the controller survives and elects a new leader from the surviving in-sync DC-B replicas. The ISR settles at the two DC-B copies (= min.ISR). Open at this step: step 4, DC-A killed - quorum holds, leadership fails over to DC-B
O3 accepted on the new DC-B leader - cluster stayed available t+12.050s A fresh acks=all write commits on the failed-over DC-B leader (ISR 2 = min.ISR) with O1/O2 intact. The witness is what made this possible: without it an even 2-voter split would have lost quorum and the election would have frozen (see 09.1.4). Open at this step: step 5, O3 accepted on the new DC-B leader - cluster stayed available

The takeaway

Surviving the loss of the leader DC needs two things at once: a controller quorum (here via a third-site witness) and a surviving in-sync replica to promote. With both, leadership fails over and writes resume with no committed-data loss.

Configs in this scenario

min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.1.5" title="Kafka Simulator: Losing a whole DC" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/losing-a-whole-dc/">Kafka data center failure: leader failover to the other DC</a></p>

More in Disaster recovery

All scenarios