---
title: "Kafka data center failure: leader failover to the other DC"
description: "Kill the DC that holds the partition leader and a KRaft voter. The dc-c witness keeps a 2-of-3 Kafka quorum, leadership fails over to DC-B and O3 commits there."
url: https://monedula.dev/kafka-simulator/scenarios/losing-a-whole-dc/
---

# Kafka data center failure: leader failover to the other DC

Scenario 09.1.5 · Losing a whole DC. Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3.

kill_dc takes out the leader DC; the witness keeps the voter quorum, so leadership fails over to the surviving DC and writes resume.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5

## What you start with

Same stretched two-DC topology as 09.1.1, but here we kill the DC that holds the partition leader and one of the controller voters. A witness voter in a third site keeps a 2-of-3 controller majority, and the surviving DC’s in-sync replicas let the controller elect a new leader there. A fresh write then commits on the new leader.

## Step by step

1. **Stretched 2-DC cluster ready - leader in DC-A** (t+0.000s): The same stretched topology as 09.1.1: RF 4 across DC-A + DC-B, min.ISR 2, a voter in each data DC plus a dc-c witness. The partition leader is one of the DC-A brokers. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5&t=0)
2. **O1 committed - durable across both DCs** (t+3.050s): p1 produces with acks=all; O1 commits only after the cross-DC ISR has it, so it is already replicated into DC-B before the producer is acked. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5&t=1)
3. **O2 committed - log {O1, O2} on all 4 replicas** (t+6.050s): A second acks=all write, fully replicated across both data centres. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5&t=2)
4. **DC-A killed - quorum holds, leadership fails over to DC-B** (t+9.600s): kill_dc takes out both DC-A brokers (leader + follower) AND the DC-A voter. c-b plus the dc-c witness are still a 2-of-3 majority, so the controller survives and elects a new leader from the surviving in-sync DC-B replicas. The ISR settles at the two DC-B copies (= min.ISR). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5&t=3)
5. **O3 accepted on the new DC-B leader - cluster stayed available** (t+12.050s): A fresh acks=all write commits on the failed-over DC-B leader (ISR 2 = min.ISR) with O1/O2 intact. The witness is what made this possible: without it an even 2-voter split would have lost quorum and the election would have frozen (see 09.1.4). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.5&t=4)

## The takeaway

Surviving the loss of the leader DC needs two things at once: a controller quorum (here via a third-site witness) and a surviving in-sync replica to promote. With both, leadership fails over and writes resume with no committed-data loss.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#replication)