---
title: "2.5-DC Kafka stretched cluster with a witness site"
description: "A DC-C witness keeps the Kafka quorum when DC-B is cut off, and Confluent Multi-Region Clusters promote the DC-A observer to restore min.insync.replicas=3."
url: https://monedula.dev/kafka-simulator/scenarios/stretched-2-5-dc/
---

# 2.5-DC Kafka stretched cluster with a witness site

Scenario 09.1.2 · Stretched 2.5-DC. Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3.

Lose a main DC; the reachable observer in the surviving DC auto-promotes into the ISR to restore min.insync.replicas.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2

## What you start with

This is Confluent’s 2.5-DC Multi-Region Cluster: two main DCs each holding sync replicas plus an async observer (RF 6), with a witness in a third half-site for the controller quorum. Observers trail asynchronously and stay out of the ISR until needed. We cut away a whole main DC so the surviving DC’s reachable observer auto-promotes into the ISR to restore min.insync.replicas.

## Step by step

1. **Stretched cluster ready - DC-A & DC-B each 2 sync + 1 observer, witness DC-C** (t+0.000s): One cluster, six brokers split 3-and-3 across DC-A and DC-B. The orders partition has 4 sync replicas (broker-1, broker-2 in DC-A, broker-4, broker-5 in DC-B) forming the ISR, plus one async observer per DC (broker-3, broker-6) that replicates but stays out of the ISR. DC-C holds only the witness controller voter c-c, giving the quorum a third vote without storing data. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2&t=0)
2. **Record committed across both DCs (ISR = 4 sync replicas ≥ min.ISR 3)** (t+5.000s): p1 writes O42 with acks=all. The high-watermark moves to 1 once all 4 sync replicas have it, so the ack already means the record is in both DCs. The observers broker-3 and broker-6 copy it later (observer lag), which never delays the ack. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2&t=1)
3. **DC-B partitioned - ISR fell below min.ISR 3; reachable DC-A observer auto-promotes** (t+8.000s): The network cuts DC-B off, and c-a + c-c still form a 2-of-3 controller majority, so broker-1 stays leader. The ISR loses broker-4 and broker-5 and would drop to 2, below min.insync.replicas 3. The reachable DC-A observer broker-3, already caught up at offset 1, is promoted into the ISR, bringing it back to 3 - this observer auto-promotion is Confluent's Multi-Region Cluster behavior, not part of Apache Kafka. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2&t=2)
4. **Surviving DC-A keeps committing - write acked on the promoted ISR of 3** (t+12.000s): p1 writes O77 while DC-B is still cut off. The ISR of 3 (broker-1, broker-2 and the promoted broker-3) meets min.ISR, so the write is acked and the high-watermark reaches 2. The DC-B replicas are still at offset 1 - without the promotion this acks=all write would have been rejected with NotEnoughReplicas. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2&t=3)
5. **Healed - DC-B sync replicas rejoin the ISR** (t+14.000s): The network heals and broker-4 and broker-5 fetch O77, reach offset 2 and rejoin the ISR. The simulator keeps broker-3 in it too (5 replicas), but real MRC would now demote broker-3 back to an observer, since the sync replicas are restored. broker-6 is still an async observer at offset 1 and catches up in the background. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.1.2&t=4)

## The takeaway

Async observers let a 2.5-DC cluster ride out the loss of an entire main DC: the survivor auto-promotes an observer back into the ISR, restoring durability without operator intervention.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Confluent documentation](https://docs.confluent.io/platform/current/multi-dc-deployments/multi-region-async/index.html)