Monedula

Scenario 10.1.1 · MRC observers

Confluent MRC observers: Kafka replicas outside the ISR

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3 · Confluent Platform feature

Confluent MRC on a 2.5-DC stretched cluster: sync replicas commit in DC-A (min.ISR 2, local low latency); an async observer in DC-B trails committed data. Witness DC-C holds the third voter.

Run this scenario in the playground

DC-B observer caught up - async replica settles (lag 0); never counted toward ISR. t+1.700s

All three committed offsets have now reached the DC-B observer; its LEO 3 equals the HW, so the lag is 0. broker-3 holds a full remote copy for disaster recovery, yet the ISR is still broker-1 and broker-2 only - the observer never added cross-DC latency to a write.

Open at this step: step 4, DC-B observer caught up - async replica settles (lag 0); never counted toward ISR

What you start with

Confluent Multi-Region Clusters mix two kinds of replica in one stretched cluster: synchronous replicas that must confirm before a write commits, and asynchronous observers that fetch from the leader like followers but never count toward the ISR. Here min.ISR-2 sync replicas commit locally in DC-A while an observer in DC-B trails the committed high-watermark and a witness in DC-C holds the third voter.

Step by step

  1. 2.5-DC ready - DC-A + DC-B hold data, witness DC-C holds the 3rd voter; observer in DC-B t+0.000s orders has two synchronous replicas in DC-A (broker-1 leader, broker-2) with min.ISR 2, and broker-3 in DC-B as an async observer outside the ISR. The controller quorum has one voter per DC, with c-c in DC-C as the witness that holds the third vote. Open at this step: step 1, 2.5-DC ready - DC-A + DC-B hold data, witness DC-C holds the 3rd voter; observer in DC-B
  2. Committed in DC-A (HW 3, min.ISR 2) - DC-B observer still at 0 (lag 3) t+1.000s O1, O2 and O3 each committed as soon as broker-2 in DC-A copied them, so the high-watermark is 3 and every acks=all ack came back without waiting on DC-B. The DC-B observer is still at LEO 0 (lag 3): observers fetch like followers but outside the ISR, and the simulator delays each observer fetch (observerLagMs 800). Open at this step: step 2, Committed in DC-A (HW 3, min.ISR 2) - DC-B observer still at 0 (lag 3)
  3. DC-B observer catching up asynchronously - trails the high-watermark (lag 2) t+1.450s The observer's first delayed fetch has landed: broker-3 is at LEO 1 while the HW stays 3, so the lag has fallen from 3 to 2. The producer is not affected - its writes were already acknowledged by the DC-A ISR. Open at this step: step 3, DC-B observer catching up asynchronously - trails the high-watermark (lag 2)
  4. DC-B observer caught up - async replica settles (lag 0); never counted toward ISR t+1.700s All three committed offsets have now reached the DC-B observer; its LEO 3 equals the HW, so the lag is 0. broker-3 holds a full remote copy for disaster recovery, yet the ISR is still broker-1 and broker-2 only - the observer never added cross-DC latency to a write. Open at this step: step 4, DC-B observer caught up - async replica settles (lag 0); never counted toward ISR

The takeaway

Observers give you a continuously updated remote copy for disaster recovery without paying cross-DC round-trip latency on every acks=all write.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.1.1" title="Kafka Simulator: MRC observers" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/mrc-observers/">Confluent MRC observers: Kafka replicas outside the ISR</a></p>

Further reading

Confluent documentation

More in Disaster recovery

All scenarios