--- title: "Confluent MRC observer promotion: losing the Kafka leader DC" description: "One Confluent MRC cluster, two outages. Lose DC-B and an observer is promoted so Kafka keeps writing; lose the leader DC and O3 fails with NOT_ENOUGH_REPLICAS." url: https://monedula.dev/kafka-simulator/scenarios/observer-promotion-reachable-leader/ --- # Confluent MRC observer promotion: losing the Kafka leader DC Scenario 10.1.4 · Observer promotion needs a reachable leader. Disaster recovery · Advanced · about 5 min · Apache Kafka 4.3 · Confluent Platform feature. Confluent MRC across an asymmetric 2.5-DC: DC-A holds the leader plus 2 sync replicas and a spare observer; DC-B holds a single sync replica and an observer; a witness DC-C breaks ties (min.ISR 3). The SAME cluster reacts two opposite ways to a DC outage. Isolate the NON-leader DC and the leader stays put - its own DC promotes an observer and durability is restored. Isolate the LEADER's DC and the partition fails over to DC-B, but the lone survivor cannot gather min.ISR even after promoting its observer, so acks=all stays blocked. The lesson: observer promotion only saves you where a reachable leader AND enough replicas survive together. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4 ## What you start with The same asymmetric 2.5-DC MRC cluster reacts two opposite ways to a DC outage. Isolate the non-leader DC and the leader's own DC promotes an observer to restore durability; isolate the leader's DC and the partition fails over to a lone survivor that cannot reach min.ISR even after promoting its observer, so acks=all stays blocked. ## Step by step 1. **Asymmetric MRC ready - DC-A: 2 sync + observer, DC-B: 1 sync + observer (min.ISR 3)** (t+0.000s): orders keeps 3 synchronous replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) plus one async observer per DC (broker-3, broker-5). The 3 sync replicas exactly meet min.ISR 3; the observers never count toward the ISR. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=0) 2. **O1 committed across the 3 sync replicas (HW 1)** (t+1.000s): The full sync ISR (broker-1, broker-2, broker-4) committed O1, so the high-watermark is 1. Both observers still trail asynchronously. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=1) 3. **CASE 1 - isolate the NON-leader DC (DC-B): leader stays, DC-A observer promotes → ISR 3** (t+2.100s): DC-B is cut away, but the c-a + c-c majority keeps the quorum and the leader (broker-1) on the DC-A side. The ISR lost DC-B's sync replica (broker-4) and fell to 2 (< min.ISR 3), so DC-A's reachable observer (broker-3) auto-promoted into the ISR - durability restored to 3 without moving the leader. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=2) 4. **Writable on the surviving DC-A side - O2 committed (HW 2)** (t+2.900s): With the promoted observer the ISR is back to 3, so the acks=all write O2 commits on the DC-A side. Promotion saved the partition because the leader's DC had a spare replica to pull in. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=3) 5. **Healed - DC-B's sync replica rejoins the ISR (back to 4)** (t+4.000s): The partition is healed: DC-B's sync replica (broker-4) refetches and re-enters the ISR. The simulator keeps broker-3 in the ISR too (4); real MRC would now demote it back to an observer, since the 3 sync replicas are restored. The leader is still broker-1 in DC-A. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=4) 6. **CASE 2 - isolate the LEADER's DC (DC-A): fails over to DC-B, observer promotes, but ISR only 2** (t+5.200s): Now DC-A (with the leader) is cut away. The c-b + c-c majority survives on the DC-B side and re-elects broker-4 as leader; DC-B's observer (broker-5) auto-promotes. But DC-B holds only that one sync replica + one observer, so the ISR reaches just 2 - short of min.ISR 3. A leader exists and an observer was promoted, yet there aren't enough replicas. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=5) 7. **acks=all is blocked - O3 rejected with NOT_ENOUGH_REPLICAS (HW stuck at 2)** (t+6.900s): The write O3 against the new DC-B leader can't be committed: with ISR 2 < min.ISR 3 the producer retries and then gives up with a terminal NOT_ENOUGH_REPLICAS. Promotion happened, but the lone survivor DC simply can't gather min.ISR - the opposite outcome to CASE 1, decided purely by WHICH DC you lost. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=6) 8. **Healed - durability restored (ISR 5), but the rejected O3 stays lost** (t+8.200s): Reconnecting DC-A makes the cluster durable again. The simulator re-expands the ISR to all 5 replicas and leaves broker-4 leading; real MRC would demote both promoted observers back to observers, and auto.leader.rebalance.enable (default true) would return leadership to the preferred replica within about 5 minutes. The O3 rejected during the outage remains permanently failed - healing does not un-reject it. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.1.4&t=7) ## The takeaway Observer promotion only rescues you where a reachable leader AND enough surviving replicas coexist - replica placement, not observer count alone, decides whether a given region loss is survivable. ## Configs in this scenario - [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/) - [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/) ## Further reading - [Confluent documentation](https://docs.confluent.io/platform/current/multi-dc-deployments/multi-region-async/index.html) ## Troubleshooting - [NotEnoughReplicasException in Kafka: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/not-enough-replicas/)