Monedula

Scenario 10.1.3 · Observer promotion

Kafka observer promotion in Confluent Multi-Region Clusters

Disaster recovery · Advanced · about 4 min · Apache Kafka 4.3 · Confluent Platform feature

Confluent MRC recovery: sync replicas in east + west (min.ISR 2) with two async observers in a dr region. Lose the west region and the in-sync count drops below min.ISR (produces stall) - so an observer catches up and is promoted into the ISR as a synchronous replica, restoring durability while the second observer stays async.

Run this scenario in the playground

New write committed on the promoted replica; one observer still async. t+1.900s

O4 committed on east + the promoted dr replica (HW 4). The remaining dr observer continues to trail asynchronously.

Open at this step: step 5, New write committed on the promoted replica; one observer still async

What you start with

When a Confluent MRC partition loses enough synchronous replicas to fall below min.ISR, an async observer can be promoted into the ISR to take over the durability role. Here sync replicas span east and west (min.ISR 2) with two DR observers; losing the west region stalls acks=all until one observer catches up and is promoted into the in-sync set.

Step by step

  1. MRC ready - sync in east + west (min.ISR 2), two async observers in dr t+0.000s orders has a synchronous replica in east and west; the two dr replicas are async observers that never count toward ISR / acks=all. Open at this step: step 1, MRC ready - sync in east + west (min.ISR 2), two async observers in dr
  2. Committed in east + west (HW 3) - dr observers trail t+0.900s Both sync replicas committed O1-O3; the high-watermark is 3. The dr observers lag (async). Open at this step: step 2, Committed in east + west (HW 3) - dr observers trail
  3. West region lost - ISR 1 < min.ISR 2 (acks=all produces stall); dr observer moved into the replica set t+1.100s broker-2 (west) died, so the in-sync count fell below min.ISR - acks=all writes would stall. The simulator moves a reachable dr observer (broker-3) into the replica set now and admits it to the ISR on the next step; in MRC, promotion is that ISR join, which happens once the observer has caught up. Open at this step: step 3, West region lost - ISR 1 < min.ISR 2 (acks=all produces stall); dr observer moved into the replica set
  4. Observer promoted to sync - ISR restored to 2, durability back t+1.400s The dr replica caught up and was promoted into the ISR; the in-sync count is back to 2. The second dr replica stays an async observer. Open at this step: step 4, Observer promoted to sync - ISR restored to 2, durability back
  5. New write committed on the promoted replica; one observer still async t+1.900s O4 committed on east + the promoted dr replica (HW 4). The remaining dr observer continues to trail asynchronously. Open at this step: step 5, New write committed on the promoted replica; one observer still async

The takeaway

Observer promotion is MRC's automatic durability backstop - it converts a trailing async copy into a counted synchronous replica so acks=all writes can resume after a region loss.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.1.3" title="Kafka Simulator: Observer promotion" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/observer-promotion/">Kafka observer promotion in Confluent Multi-Region Clusters</a></p>

Further reading

Confluent documentation

More in Disaster recovery

All scenarios