Monedula

Scenario 09.1.3 · Stretched 3-DC

3-DC Kafka cluster: the KRaft quorum survives losing a site

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3

Lose one of three DCs; a 2-of-3 voter majority keeps elections alive.

Run this scenario in the playground

DC-A partitioned - 2-of-3 voters keep quorum, leadership moves to DC-B. t+8.000s

DC-A is cut off. c-b and c-c still form a 2-of-3 voter majority and elect c-b as the new active controller, so metadata changes keep flowing. broker-1 drops out of every ISR and partition 0 moves its leadership from broker-1 to broker-2 (leader epoch 1). Each ISR still has 2 members, which meets min.insync.replicas 2, so all three partitions stay writable.

Open at this step: step 3, DC-A partitioned - 2-of-3 voters keep quorum, leadership moves to DC-B

What you start with

A single cluster is stretched across three data centres, one KRaft controller voter in each. With three voters, any single DC failure still leaves a 2-of-3 majority. We isolate one DC and watch the surviving two retain the controller quorum so leader elections keep running, then heal the network.

Step by step

  1. Three-DC stretched cluster ready t+0.000s One cluster with a broker and a KRaft controller voter in each of three DCs (broker-1 / c-a in DC-A, broker-2 / c-b in DC-B, broker-3 / c-c in DC-C). orders has 3 partitions with RF 3, so every partition has a replica in every DC, and leadership is spread one partition per broker. c-a is the active controller. Open at this step: step 1, Three-DC stretched cluster ready
  2. Record committed across all three DCs t+5.000s p1 writes O42 with acks=all and it lands on partition 1, led by broker-2. The high-watermark advances to 1 only after all three ISR members have it, so the acked record is already stored in all three DCs. Open at this step: step 2, Record committed across all three DCs
  3. DC-A partitioned - 2-of-3 voters keep quorum, leadership moves to DC-B t+8.000s DC-A is cut off. c-b and c-c still form a 2-of-3 voter majority and elect c-b as the new active controller, so metadata changes keep flowing. broker-1 drops out of every ISR and partition 0 moves its leadership from broker-1 to broker-2 (leader epoch 1). Each ISR still has 2 members, which meets min.insync.replicas 2, so all three partitions stay writable. Open at this step: step 3, DC-A partitioned - 2-of-3 voters keep quorum, leadership moves to DC-B
  4. Healed - DC-A rejoins, controller returns to c-a t+14.000s The network heals and broker-1 rejoins the ISR of all three partitions, with nothing to catch up on. The simulator makes c-a the active controller again here; real KRaft has no preferred controller, so c-b would keep the role and c-a would rejoin as a follower. Partition 0 stays led by broker-2 - leadership only returns to its preferred replica broker-1 when a preferred-leader election runs. Open at this step: step 4, Healed - DC-A rejoins, controller returns to c-a

The takeaway

Three DCs give the control plane a true majority quorum: losing any one DC still leaves 2-of-3 voters, so elections and metadata updates never freeze.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.1.3" title="Kafka Simulator: Stretched 3-DC" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/stretched-3-dc/">3-DC Kafka cluster: the KRaft quorum survives losing a site</a></p>

Further reading

Apache Kafka documentation

More in Disaster recovery

All scenarios