Monedula

Scenario 10.1.2 · Observer lag and async replication

Kafka observer lag in Confluent Multi-Region Clusters

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3 · Confluent Platform feature

Confluent MRC: the sync ISR in DC-A commits a steady write stream (HW races ahead) while the DC-B observer replicates ASYNCHRONOUSLY and trails. Watch the observer lag - high-watermark minus the observer LEO - build under load and then drain once the burst ends, all without the observer ever counting toward the ISR or acks=all.

Run this scenario in the playground

Burst committed in DC-A (HW 5) - observer still at 0 (lag 5). t+1.100s

The DC-A sync ISR committed O1-O5 the moment each arrived, so the high-watermark is already 5. The observer's async fetches (observerLagMs 1000) have not landed yet - observer LEO 0, lag = HW − observer LEO = 5.

Open at this step: step 2, Burst committed in DC-A (HW 5) - observer still at 0 (lag 5)

What you start with

This scenario tracks observer lag - the committed high-watermark minus the observer's log-end offset - on a Confluent MRC cluster. A steady write stream commits through the DC-A sync ISR while the DC-B observer replicates asynchronously, so it falls behind under load and then catches back up once the burst ends.

Step by step

  1. MRC ready - 2 sync replicas in DC-A (min.ISR 2), one async observer in DC-B t+0.000s orders has two synchronous replicas in DC-A and one async observer in DC-B. The observer fetches lazily and never counts toward the ISR or acks=all. Open at this step: step 1, MRC ready - 2 sync replicas in DC-A (min.ISR 2), one async observer in DC-B
  2. Burst committed in DC-A (HW 5) - observer still at 0 (lag 5) t+1.100s The DC-A sync ISR committed O1-O5 the moment each arrived, so the high-watermark is already 5. The observer's async fetches (observerLagMs 1000) have not landed yet - observer LEO 0, lag = HW − observer LEO = 5. Open at this step: step 2, Burst committed in DC-A (HW 5) - observer still at 0 (lag 5)
  3. Observer draining asynchronously - lag shrinking (HW 5, observer LEO 2) t+1.750s The observer is catching up one delayed fetch at a time: it has reached offset 2 while the HW stays 5, so the lag has fallen from 5 to 3. The observer is still outside the ISR. Open at this step: step 3, Observer draining asynchronously - lag shrinking (HW 5, observer LEO 2)
  4. Observer caught up - async lag back to 0; never entered the ISR t+2.100s All five committed offsets have now reached the DC-B observer; observer LEO 5 equals the HW, so the lag is 0 again. The observer carried a full copy without ever being part of the synchronous quorum. Open at this step: step 4, Observer caught up - async lag back to 0; never entered the ISR

The takeaway

An observer's lag is normal and self-correcting: because it never counts toward the ISR or acks=all, it absorbs load spikes by trailing and draining rather than ever slowing producers down.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.1.2" title="Kafka Simulator: Observer lag and async replication" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/observer-lag/">Kafka observer lag in Confluent Multi-Region Clusters</a></p>

Further reading

Confluent documentation

More in Disaster recovery

All scenarios