Scenario 10.1.4 · Observer promotion needs a reachable leader
Confluent MRC observer promotion: losing the Kafka leader DC
Confluent MRC across an asymmetric 2.5-DC: DC-A holds the leader plus 2 sync replicas and a spare observer; DC-B holds a single sync replica and an observer; a witness DC-C breaks ties (min.ISR 3). The SAME cluster reacts two opposite ways to a DC outage. Isolate the NON-leader DC and the leader stays put - its own DC promotes an observer and durability is restored. Isolate the LEADER's DC and the partition fails over to DC-B, but the lone survivor cannot gather min.ISR even after promoting its observer, so acks=all stays blocked. The lesson: observer promotion only saves you where a reachable leader AND enough replicas survive together.
Run this scenario in the playgroundAsymmetric MRC ready - DC-A: 2 sync + observer, DC-B: 1 sync + observer (min.ISR 3). t+0.000s
orders keeps 3 synchronous replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) plus one async observer per DC (broker-3, broker-5). The 3 sync replicas exactly meet min.ISR 3; the observers never count toward the ISR.
Open at this step: step 1, Asymmetric MRC ready - DC-A: 2 sync + observer, DC-B: 1 sync + observer (min.ISR 3)O1 committed across the 3 sync replicas (HW 1). t+1.000s
The full sync ISR (broker-1, broker-2, broker-4) committed O1, so the high-watermark is 1. Both observers still trail asynchronously.
Open at this step: step 2, O1 committed across the 3 sync replicas (HW 1)CASE 1 - isolate the NON-leader DC (DC-B): leader stays, DC-A observer promotes → ISR 3. t+2.100s
DC-B is cut away, but the c-a + c-c majority keeps the quorum and the leader (broker-1) on the DC-A side. The ISR lost DC-B's sync replica (broker-4) and fell to 2 (< min.ISR 3), so DC-A's reachable observer (broker-3) auto-promoted into the ISR - durability restored to 3 without moving the leader.
Open at this step: step 3, CASE 1 - isolate the NON-leader DC (DC-B): leader stays, DC-A observer promotes → ISR 3Writable on the surviving DC-A side - O2 committed (HW 2). t+2.900s
With the promoted observer the ISR is back to 3, so the acks=all write O2 commits on the DC-A side. Promotion saved the partition because the leader's DC had a spare replica to pull in.
Open at this step: step 4, Writable on the surviving DC-A side - O2 committed (HW 2)Healed - DC-B's sync replica rejoins the ISR (back to 4). t+4.000s
The partition is healed: DC-B's sync replica (broker-4) refetches and re-enters the ISR. The simulator keeps broker-3 in the ISR too (4); real MRC would now demote it back to an observer, since the 3 sync replicas are restored. The leader is still broker-1 in DC-A.
Open at this step: step 5, Healed - DC-B's sync replica rejoins the ISR (back to 4)CASE 2 - isolate the LEADER's DC (DC-A): fails over to DC-B, observer promotes, but ISR only 2. t+5.200s
Now DC-A (with the leader) is cut away. The c-b + c-c majority survives on the DC-B side and re-elects broker-4 as leader; DC-B's observer (broker-5) auto-promotes. But DC-B holds only that one sync replica + one observer, so the ISR reaches just 2 - short of min.ISR 3. A leader exists and an observer was promoted, yet there aren't enough replicas.
Open at this step: step 6, CASE 2 - isolate the LEADER's DC (DC-A): fails over to DC-B, observer promotes, but ISR only 2acks=all is blocked - O3 rejected with NOT_ENOUGH_REPLICAS (HW stuck at 2). t+6.900s
The write O3 against the new DC-B leader can't be committed: with ISR 2 < min.ISR 3 the producer retries and then gives up with a terminal NOT_ENOUGH_REPLICAS. Promotion happened, but the lone survivor DC simply can't gather min.ISR - the opposite outcome to CASE 1, decided purely by WHICH DC you lost.
Open at this step: step 7, acks=all is blocked - O3 rejected with NOT_ENOUGH_REPLICAS (HW stuck at 2)Healed - durability restored (ISR 5), but the rejected O3 stays lost. t+8.200s
Reconnecting DC-A makes the cluster durable again. The simulator re-expands the ISR to all 5 replicas and leaves broker-4 leading; real MRC would demote both promoted observers back to observers, and auto.leader.rebalance.enable (default true) would return leadership to the preferred replica within about 5 minutes. The O3 rejected during the outage remains permanently failed - healing does not un-reject it.
Open at this step: step 8, Healed - durability restored (ISR 5), but the rejected O3 stays lostWhat you start with
The same asymmetric 2.5-DC MRC cluster reacts two opposite ways to a DC outage. Isolate the non-leader DC and the leader's own DC promotes an observer to restore durability; isolate the leader's DC and the partition fails over to a lone survivor that cannot reach min.ISR even after promoting its observer, so acks=all stays blocked.
Step by step
- Asymmetric MRC ready - DC-A: 2 sync + observer, DC-B: 1 sync + observer (min.ISR 3) t+0.000s orders keeps 3 synchronous replicas (broker-1 leader + broker-2 in DC-A, broker-4 in DC-B) plus one async observer per DC (broker-3, broker-5). The 3 sync replicas exactly meet min.ISR 3; the observers never count toward the ISR. Open at this step: step 1, Asymmetric MRC ready - DC-A: 2 sync + observer, DC-B: 1 sync + observer (min.ISR 3)
- O1 committed across the 3 sync replicas (HW 1) t+1.000s The full sync ISR (broker-1, broker-2, broker-4) committed O1, so the high-watermark is 1. Both observers still trail asynchronously. Open at this step: step 2, O1 committed across the 3 sync replicas (HW 1)
- CASE 1 - isolate the NON-leader DC (DC-B): leader stays, DC-A observer promotes → ISR 3 t+2.100s DC-B is cut away, but the c-a + c-c majority keeps the quorum and the leader (broker-1) on the DC-A side. The ISR lost DC-B's sync replica (broker-4) and fell to 2 (< min.ISR 3), so DC-A's reachable observer (broker-3) auto-promoted into the ISR - durability restored to 3 without moving the leader. Open at this step: step 3, CASE 1 - isolate the NON-leader DC (DC-B): leader stays, DC-A observer promotes → ISR 3
- Writable on the surviving DC-A side - O2 committed (HW 2) t+2.900s With the promoted observer the ISR is back to 3, so the acks=all write O2 commits on the DC-A side. Promotion saved the partition because the leader's DC had a spare replica to pull in. Open at this step: step 4, Writable on the surviving DC-A side - O2 committed (HW 2)
- Healed - DC-B's sync replica rejoins the ISR (back to 4) t+4.000s The partition is healed: DC-B's sync replica (broker-4) refetches and re-enters the ISR. The simulator keeps broker-3 in the ISR too (4); real MRC would now demote it back to an observer, since the 3 sync replicas are restored. The leader is still broker-1 in DC-A. Open at this step: step 5, Healed - DC-B's sync replica rejoins the ISR (back to 4)
- CASE 2 - isolate the LEADER's DC (DC-A): fails over to DC-B, observer promotes, but ISR only 2 t+5.200s Now DC-A (with the leader) is cut away. The c-b + c-c majority survives on the DC-B side and re-elects broker-4 as leader; DC-B's observer (broker-5) auto-promotes. But DC-B holds only that one sync replica + one observer, so the ISR reaches just 2 - short of min.ISR 3. A leader exists and an observer was promoted, yet there aren't enough replicas. Open at this step: step 6, CASE 2 - isolate the LEADER's DC (DC-A): fails over to DC-B, observer promotes, but ISR only 2
- acks=all is blocked - O3 rejected with NOT_ENOUGH_REPLICAS (HW stuck at 2) t+6.900s The write O3 against the new DC-B leader can't be committed: with ISR 2 < min.ISR 3 the producer retries and then gives up with a terminal NOT_ENOUGH_REPLICAS. Promotion happened, but the lone survivor DC simply can't gather min.ISR - the opposite outcome to CASE 1, decided purely by WHICH DC you lost. Open at this step: step 7, acks=all is blocked - O3 rejected with NOT_ENOUGH_REPLICAS (HW stuck at 2)
- Healed - durability restored (ISR 5), but the rejected O3 stays lost t+8.200s Reconnecting DC-A makes the cluster durable again. The simulator re-expands the ISR to all 5 replicas and leaves broker-4 leading; real MRC would demote both promoted observers back to observers, and auto.leader.rebalance.enable (default true) would return leadership to the preferred replica within about 5 minutes. The O3 rejected during the outage remains permanently failed - healing does not un-reject it. Open at this step: step 8, Healed - durability restored (ISR 5), but the rejected O3 stays lost
The takeaway
Observer promotion only rescues you where a reachable leader AND enough surviving replicas coexist - replica placement, not observer count alone, decides whether a given region loss is survivable.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.1.4" title="Kafka Simulator: Observer promotion needs a reachable leader" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/observer-promotion-reachable-leader/">Confluent MRC observer promotion: losing the Kafka leader DC</a></p>Further reading
Troubleshooting
More in Disaster recovery
- Kafka stretched cluster across two data centers
- 2.5-DC Kafka stretched cluster with a witness site
- 3-DC Kafka cluster: the KRaft quorum survives losing a site
- Kafka in two DCs without a witness: KRaft quorum is lost
- Kafka data center failure: leader failover to the other DC
- Kafka network split inside one DC: ISR drops to the floor
- Kafka MirrorMaker 2 active-passive DR and replication lag
- Kafka DR failover with MirrorMaker 2: measuring RPO
- Active-active Kafka with MirrorMaker 2 and region topics
- MirrorMaker 2 offset translation for Kafka consumer failover
- Kafka failback after a MirrorMaker 2 DR failover
- Active-active Kafka DR drill: move clients to the survivor
- Confluent MRC observers: Kafka replicas outside the ISR
- Kafka observer lag in Confluent Multi-Region Clusters
- Kafka observer promotion in Confluent Multi-Region Clusters
- Kafka replica placement: durability vs latency across DCs
- Confluent Cluster Linking: offset-preserving Kafka mirroring
- Kafka mirror topics (Confluent): read-only until promoted
- Cluster Linking vs MirrorMaker 2 for Kafka replication
- Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2
- Kafka failover and failback with Confluent Cluster Linking