--- title: "Zero data loss through a Kafka DC outage with Confluent MRC" description: "With min.insync.replicas=3 across two DCs, cutting off DC-B makes Confluent Multi-Region Clusters promote the DC-A observer, so Kafka writes never stop." url: https://monedula.dev/kafka-simulator/scenarios/survive-regional-outage-zero-loss/ --- # Zero data loss through a Kafka DC outage with Confluent MRC Scenario 12.0.2 · Capstone: survive a regional outage with zero data loss. Capstones · Advanced · about 4 min · Apache Kafka 4.3 · Confluent Platform feature. MRC with min.ISR=3 across two DCs plus spare observers: lose a whole DC and the surviving DC auto-promotes its observer back into the ISR, so durability is restored and writes never stop - no committed record lost. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2 ## What you start with A Confluent Multi-Region Cluster tuned to survive a full DC loss with zero data loss. min.insync.replicas=3 with a sync placement of 2 replicas in DC-A + 1 in DC-B forces every acks=all write across two DCs, and each DC also holds an async observer that trails the ISR but never counts toward min.ISR. Isolate the non-leader DC and watch the surviving DC auto-promote its observer back into the ISR. ## Step by step 1. **MRC ready - min.ISR=3 across two DCs** (t+0.000s): payments has 3 sync replicas (2 in DC-A, 1 in DC-B) plus an async observer in each DC, min.insync.replicas=3. A committed write must reach all three sync replicas across both DCs. DC-C holds a witness controller voter. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=0) 2. **O1 committed - ISR = 3** (t+0.800s): The acks=all write O1 is confirmed only once all three sync replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) have it. The observers trail asynchronously and are not in the ISR. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=1) 3. **DC-B lost - DC-A observer auto-promotes** (t+2.000s): DC-B is isolated. The c-a + witness c-c majority keeps the quorum and leadership in DC-A, so the leader stays reachable. The ISR loses broker-4 and drops to 2 (< min.ISR 3); DC-A's reachable observer (broker-3) auto-promotes into the ISR - durability restored to 3. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=2) 4. **O2 committed on the survivor - still writable** (t+2.500s): With the promoted observer the ISR is 3 again, so the acks=all write O2 commits WITHOUT DC-B. The cluster never went read-only and no committed record was lost - zero-RPO survival of a full DC outage. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=3) 5. **Healed - DC-B rejoins the ISR** (t+3.500s): DC-B reconnects; broker-4 refetches the records it missed and re-enters the ISR. Durability is back to its full cross-DC footprint. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=4) 6. **O3 committed - fully restored** (t+4.200s): O3 commits against the healed cluster, confirming writes flow normally again across both DCs. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.2&t=5) ## The takeaway Zero-data-loss survival of a regional outage takes a cross-DC sync replica, a high enough min.ISR to prove cross-DC durability, AND a spare observer the survivor can promote - only then do writes never stop and no committed record is lost. ## Configs in this scenario - [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/) - [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/) ## Further reading - [Confluent documentation](https://docs.confluent.io/platform/current/multi-dc-deployments/multi-region-async/index.html)