Monedula

Scenario 09.1.1 · Stretched cluster across 2 DCs

Kafka stretched cluster across two data centers

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3

One cluster, RF spread across two DCs: while the ISR spans both sites, an acks=all write is durable in both, so it survives losing a whole DC.

Run this scenario in the playground

O3 accepted - the acked log survived losing a DC. t+12.050s

A fresh acks=all write still commits on the DC-A leader (ISR 2 = min.ISR), and O1/O2 are intact. Because the write had been durable in both DCs, losing one cost no acknowledged data.

Open at this step: step 5, O3 accepted - the acked log survived losing a DC

What you start with

A single logical Kafka cluster is stretched across two data centres, with an acks=all partition’s RF-4 replicas placed evenly so the in-sync replica set (ISR) spans both sites. Because a committed write must reach the ISR before the producer is acked, every record is durable in both DCs the instant it commits, as long as the ISR spans both sites. We then lose the follower DC to watch the ISR shrink to the min.insync.replicas floor while a fresh write still commits.

Step by step

  1. Stretched 2-DC cluster ready - RF 4 across DC-A + DC-B t+0.000s One logical cluster, brokers split 2-and-2 across DC-A and DC-B. The orders partition has RF 4, so a replica lives on every broker - the replica set spans both data centres. A controller voter sits in each data DC plus a witness in dc-c, keeping the quorum odd. Open at this step: step 1, Stretched 2-DC cluster ready - RF 4 across DC-A + DC-B
  2. O1 committed - leader in DC-A, ISR spans both DCs t+3.050s p1 produces with acks=all. The record only commits once the full ISR has it, and the ISR includes replicas in BOTH DCs - so the moment p1 is acked, O1 is already durable in DC-A and DC-B. Open at this step: step 2, O1 committed - leader in DC-A, ISR spans both DCs
  3. O2 committed - log {O1, O2} on all 4 replicas t+6.050s A second acks=all write. Both records are now replicated cross-DC to all four replicas; the leader's high-watermark has advanced past both. Open at this step: step 3, O2 committed - log {O1, O2} on all 4 replicas
  4. DC-B lost - ISR shrinks to the 2 DC-A replicas (= min.ISR) t+9.050s kill_dc drops both DC-B brokers. The leader sits in DC-A so leadership is untouched; the ISR just sheds the two DC-B copies, settling at the 2 DC-A replicas - exactly min.insync.replicas, so the partition is still writable. Open at this step: step 4, DC-B lost - ISR shrinks to the 2 DC-A replicas (= min.ISR)
  5. O3 accepted - the acked log survived losing a DC t+12.050s A fresh acks=all write still commits on the DC-A leader (ISR 2 = min.ISR), and O1/O2 are intact. Because the write had been durable in both DCs, losing one cost no acknowledged data. Open at this step: step 5, O3 accepted - the acked log survived losing a DC

The takeaway

Stretching one cluster across two DCs gives synchronous cross-site durability while the ISR spans both sites: acked data already exists in both, so losing a DC costs no committed records. Only a min.insync.replicas above the replicas in one DC (3 here) guarantees it; at 2, an ISR shrunk to DC-A keeps acking single-site writes like O3.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.1.1" title="Kafka Simulator: Stretched cluster across 2 DCs" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/stretched-cluster-2-dc/">Kafka stretched cluster across two data centers</a></p>

Further reading

Apache Kafka documentation

More in Disaster recovery

All scenarios