Scenario 09.1.2 · Stretched 2.5-DC

2.5-DC Kafka stretched cluster with a witness site

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3

Lose a main DC; the reachable observer in the surviving DC auto-promotes into the ISR to restore min.insync.replicas.

What you start with

This is Confluent’s 2.5-DC Multi-Region Cluster: two main DCs each holding sync replicas plus an async observer (RF 6), with a witness in a third half-site for the controller quorum. Observers trail asynchronously and stay out of the ISR until needed. We cut away a whole main DC so the surviving DC’s reachable observer auto-promotes into the ISR to restore min.insync.replicas.

Step by step

1. Stretched cluster ready - DC-A & DC-B each 2 sync + 1 observer, witness DC-C t+0.000s One cluster, six brokers split 3-and-3 across DC-A and DC-B. The orders partition has 4 sync replicas (broker-1, broker-2 in DC-A, broker-4, broker-5 in DC-B) forming the ISR, plus one async observer per DC (broker-3, broker-6) that replicates but stays out of the ISR. DC-C holds only the witness controller voter c-c, giving the quorum a third vote without storing data. Open at this step: step 1, Stretched cluster ready - DC-A & DC-B each 2 sync + 1 observer, witness DC-C
2. Record committed across both DCs (ISR = 4 sync replicas ≥ min.ISR 3) t+5.000s p1 writes O42 with acks=all. The high-watermark moves to 1 once all 4 sync replicas have it, so the ack already means the record is in both DCs. The observers broker-3 and broker-6 copy it later (observer lag), which never delays the ack. Open at this step: step 2, Record committed across both DCs (ISR = 4 sync replicas ≥ min.ISR 3)
3. DC-B partitioned - ISR fell below min.ISR 3; reachable DC-A observer auto-promotes t+8.000s The network cuts DC-B off, and c-a + c-c still form a 2-of-3 controller majority, so broker-1 stays leader. The ISR loses broker-4 and broker-5 and would drop to 2, below min.insync.replicas 3. The reachable DC-A observer broker-3, already caught up at offset 1, is promoted into the ISR, bringing it back to 3 - this observer auto-promotion is Confluent's Multi-Region Cluster behavior, not part of Apache Kafka. Open at this step: step 3, DC-B partitioned - ISR fell below min.ISR 3; reachable DC-A observer auto-promotes
4. Surviving DC-A keeps committing - write acked on the promoted ISR of 3 t+12.000s p1 writes O77 while DC-B is still cut off. The ISR of 3 (broker-1, broker-2 and the promoted broker-3) meets min.ISR, so the write is acked and the high-watermark reaches 2. The DC-B replicas are still at offset 1 - without the promotion this acks=all write would have been rejected with NotEnoughReplicas. Open at this step: step 4, Surviving DC-A keeps committing - write acked on the promoted ISR of 3
5. Healed - DC-B sync replicas rejoin the ISR t+14.000s The network heals and broker-4 and broker-5 fetch O77, reach offset 2 and rejoin the ISR. The simulator keeps broker-3 in it too (5 replicas), but real MRC would now demote broker-3 back to an observer, since the sync replicas are restored. broker-6 is still an async observer at offset 1 and catches up in the background. Open at this step: step 5, Healed - DC-B sync replicas rejoin the ISR

The takeaway

Async observers let a 2.5-DC cluster ride out the loss of an entire main DC: the survivor auto-promotes an observer back into the ISR, restoring durability without operator intervention.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.1.2" title="Kafka Simulator: Stretched 2.5-DC" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/stretched-2-5-dc/">2.5-DC Kafka stretched cluster with a witness site</a></p>

Further reading

Confluent documentation

More in Disaster recovery

• Kafka stretched cluster across two data centers
• 3-DC Kafka cluster: the KRaft quorum survives losing a site
• Kafka in two DCs without a witness: KRaft quorum is lost
• Kafka data center failure: leader failover to the other DC
• Kafka network split inside one DC: ISR drops to the floor
• Kafka MirrorMaker 2 active-passive DR and replication lag
• Kafka DR failover with MirrorMaker 2: measuring RPO
• Active-active Kafka with MirrorMaker 2 and region topics
• MirrorMaker 2 offset translation for Kafka consumer failover
• Kafka failback after a MirrorMaker 2 DR failover
• Active-active Kafka DR drill: move clients to the survivor
• Confluent MRC observers: Kafka replicas outside the ISR
• Kafka observer lag in Confluent Multi-Region Clusters
• Kafka observer promotion in Confluent Multi-Region Clusters
• Confluent MRC observer promotion: losing the Kafka leader DC
• Kafka replica placement: durability vs latency across DCs
• Confluent Cluster Linking: offset-preserving Kafka mirroring
• Kafka mirror topics (Confluent): read-only until promoted
• Cluster Linking vs MirrorMaker 2 for Kafka replication
• Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2
• Kafka failover and failback with Confluent Cluster Linking

All scenarios