Monedula

Scenario 10.2.4 · Consumer offset sync across a link

Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2

Disaster recovery · Advanced · about 4 min · Apache Kafka 4.3

A cluster link syncs committed consumer offsets AS-IS (committed === translated) because offsets are preserved; MM2 instead translates them through its offsetMap. Both sync on a cadence, so commits after the last sync are reprocessed either way.

Run this scenario in the playground

Offsets synced over the links. t+1.100s

The checkpoint at t=1000 syncs both groups. CLUSTER LINK cl-orders: committed === translated (3 === 3) - offsets are preserved, so the synced offset IS the committed offset, with no translation. MM2 mm-payments: the committed offset is run through the offsetMap translation. On both links a failover replays whatever was committed after the last sync, so the reprocess window is set by the sync cadence (see 09.2.4).

Open at this step: step 3, Offsets synced over the links

What you start with

Cluster Linking can sync committed consumer-group offsets across the link. Because the mirror preserves source offsets exactly, a synced offset arrives as-is (committed === translated) and a consumer resumes at the same record; MirrorMaker 2 instead translates offsets through its offsetMap at the checkpoint cadence.

Step by step

  1. Two links, two consumers t+0.000s g1 reads + commits on orders (mirrored by a CLUSTER LINK); g2 reads + commits on payments (mirrored by MM2). Both links checkpoint consumer offsets at the same cadence (mirrorCheckpointMs), so the only difference will be how each translates the committed offset. Open at this step: step 1, Two links, two consumers
  2. Both groups committed t+0.800s Each group has drained its three records and committed offset 3 ({0:3}). The mirrors are caught up too, so the destination high-watermarks match the source. Nothing is synced across the links yet - that waits for the checkpoint tick. Open at this step: step 2, Both groups committed
  3. Offsets synced over the links t+1.100s The checkpoint at t=1000 syncs both groups. CLUSTER LINK cl-orders: committed === translated (3 === 3) - offsets are preserved, so the synced offset IS the committed offset, with no translation. MM2 mm-payments: the committed offset is run through the offsetMap translation. On both links a failover replays whatever was committed after the last sync, so the reprocess window is set by the sync cadence (see 09.2.4). Open at this step: step 3, Offsets synced over the links

The takeaway

Offset-preserving sync means a failed-over consumer resumes exactly at its last synced offset with no translation, whereas MM2's translated checkpoints can land early; on both, anything committed after the last sync is reprocessed.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.2.4" title="Kafka Simulator: Consumer offset sync across a link" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/consumer-offset-sync-link/">Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2</a></p>

Further reading

Confluent documentation

More in Disaster recovery

All scenarios