Scenario 12.0.2 · Capstone: survive a regional outage with zero data loss
Zero data loss through a Kafka DC outage with Confluent MRC
MRC with min.ISR=3 across two DCs plus spare observers: lose a whole DC and the surviving DC auto-promotes its observer back into the ISR, so durability is restored and writes never stop - no committed record lost.
Run this scenario in the playgroundMRC ready - min.ISR=3 across two DCs. t+0.000s
payments has 3 sync replicas (2 in DC-A, 1 in DC-B) plus an async observer in each DC, min.insync.replicas=3. A committed write must reach all three sync replicas across both DCs. DC-C holds a witness controller voter.
Open at this step: step 1, MRC ready - min.ISR=3 across two DCsO1 committed - ISR = 3. t+0.800s
The acks=all write O1 is confirmed only once all three sync replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) have it. The observers trail asynchronously and are not in the ISR.
Open at this step: step 2, O1 committed - ISR = 3DC-B lost - DC-A observer auto-promotes. t+2.000s
DC-B is isolated. The c-a + witness c-c majority keeps the quorum and leadership in DC-A, so the leader stays reachable. The ISR loses broker-4 and drops to 2 (< min.ISR 3); DC-A's reachable observer (broker-3) auto-promotes into the ISR - durability restored to 3.
Open at this step: step 3, DC-B lost - DC-A observer auto-promotesO2 committed on the survivor - still writable. t+2.500s
With the promoted observer the ISR is 3 again, so the acks=all write O2 commits WITHOUT DC-B. The cluster never went read-only and no committed record was lost - zero-RPO survival of a full DC outage.
Open at this step: step 4, O2 committed on the survivor - still writableHealed - DC-B rejoins the ISR. t+3.500s
DC-B reconnects; broker-4 refetches the records it missed and re-enters the ISR. Durability is back to its full cross-DC footprint.
Open at this step: step 5, Healed - DC-B rejoins the ISRO3 committed - fully restored. t+4.200s
O3 commits against the healed cluster, confirming writes flow normally again across both DCs.
Open at this step: step 6, O3 committed - fully restoredWhat you start with
A Confluent Multi-Region Cluster tuned to survive a full DC loss with zero data loss. min.insync.replicas=3 with a sync placement of 2 replicas in DC-A + 1 in DC-B forces every acks=all write across two DCs, and each DC also holds an async observer that trails the ISR but never counts toward min.ISR. Isolate the non-leader DC and watch the surviving DC auto-promote its observer back into the ISR.
Step by step
- MRC ready - min.ISR=3 across two DCs t+0.000s payments has 3 sync replicas (2 in DC-A, 1 in DC-B) plus an async observer in each DC, min.insync.replicas=3. A committed write must reach all three sync replicas across both DCs. DC-C holds a witness controller voter. Open at this step: step 1, MRC ready - min.ISR=3 across two DCs
- O1 committed - ISR = 3 t+0.800s The acks=all write O1 is confirmed only once all three sync replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) have it. The observers trail asynchronously and are not in the ISR. Open at this step: step 2, O1 committed - ISR = 3
- DC-B lost - DC-A observer auto-promotes t+2.000s DC-B is isolated. The c-a + witness c-c majority keeps the quorum and leadership in DC-A, so the leader stays reachable. The ISR loses broker-4 and drops to 2 (< min.ISR 3); DC-A's reachable observer (broker-3) auto-promotes into the ISR - durability restored to 3. Open at this step: step 3, DC-B lost - DC-A observer auto-promotes
- O2 committed on the survivor - still writable t+2.500s With the promoted observer the ISR is 3 again, so the acks=all write O2 commits WITHOUT DC-B. The cluster never went read-only and no committed record was lost - zero-RPO survival of a full DC outage. Open at this step: step 4, O2 committed on the survivor - still writable
- Healed - DC-B rejoins the ISR t+3.500s DC-B reconnects; broker-4 refetches the records it missed and re-enters the ISR. Durability is back to its full cross-DC footprint. Open at this step: step 5, Healed - DC-B rejoins the ISR
- O3 committed - fully restored t+4.200s O3 commits against the healed cluster, confirming writes flow normally again across both DCs. Open at this step: step 6, O3 committed - fully restored
The takeaway
Zero-data-loss survival of a regional outage takes a cross-DC sync replica, a high enough min.ISR to prove cross-DC durability, AND a spare observer the survivor can promote - only then do writes never stop and no committed record is lost.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=12.0.2" title="Kafka Simulator: Capstone: survive a regional outage with zero data loss" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/survive-regional-outage-zero-loss/">Zero data loss through a Kafka DC outage with Confluent MRC</a></p>