Monedula

Scenario 09.2.4 · Offset translation across clusters

MirrorMaker 2 offset translation for Kafka consumer failover

Disaster recovery · Advanced · about 4 min · Apache Kafka 4.3

MM2 checkpoints the committed offset, translated to the DR cluster. Fail over and the consumer resumes at the last checkpoint - reprocessing the un-checkpointed tail.

Run this scenario in the playground

Failover: consumer resumes at the checkpoint (offset 3), not 5. t+1.810s

g1 is promoted onto dr's primary.orders. Offset translation resumes it at the last checkpointed offset (3), NOT its live committed (5) - so records 3,4 will be reprocessed. RPO is still 0 (all data mirrored); the only cost is reprocessing the un-checkpointed tail.

Open at this step: step 4, Failover: consumer resumes at the checkpoint (offset 3), not 5

What you start with

When MirrorMaker 2 re-produces records into the DR cluster, the DR offsets can drift from the source offsets and the group's commits stay on the source, so a failed-over consumer cannot reuse its primary offsets directly. MM2’s checkpoint connector periodically records the group’s committed offset translated into the DR offset space. The consumer keeps committing after the last checkpoint; on failover it resumes at that checkpoint, reprocessing the un-checkpointed tail.

Step by step

  1. Two regions ready - consumer reads on primary, MM2 mirrors to dr t+0.000s primary serves orders; the mm-1 link mirrors committed records into dr's MM2 topic primary.orders. Checkpoints run every 1000 ms here (real MM2: emit.checkpoints.interval.seconds, 60 s by default): MM2 will periodically checkpoint g1's committed offset, translated into dr's offset space. Open at this step: step 1, Two regions ready - consumer reads on primary, MM2 mirrors to dr
  2. First checkpoint: committed offset 3 translated to dr t+1.100s g1 has consumed and committed offsets 0,1,2 (committed = 3). The t=1000 checkpoint records that translated offset on the link (open the mm-1 link inspector → offset checkpoints). The mirror has kept up so RPO is 0. Open at this step: step 2, First checkpoint: committed offset 3 translated to dr
  3. Consumer commits past the checkpoint - reprocess window opens t+1.600s p1 produced O4,O5; g1 commits up to 5. But the next checkpoint isn't until t=2000, so offsets 3,4 are committed-but-not-checkpointed. Fail over now and the consumer would re-read those 2 records - the reprocess window. Open at this step: step 3, Consumer commits past the checkpoint - reprocess window opens
  4. Failover: consumer resumes at the checkpoint (offset 3), not 5 t+1.810s g1 is promoted onto dr's primary.orders. Offset translation resumes it at the last checkpointed offset (3), NOT its live committed (5) - so records 3,4 will be reprocessed. RPO is still 0 (all data mirrored); the only cost is reprocessing the un-checkpointed tail. Open at this step: step 4, Failover: consumer resumes at the checkpoint (offset 3), not 5
  5. Consumer catches up on dr; a new produce lands on the promoted region t+2.400s g1 re-reads offsets 3,4 then the new O6 (produced to dr's primary.orders after promotion), draining back to the dr high-watermark. Writing the mirror topic is a simulator shortcut: MM2 expects producers to write only local topics, so a real p1 would write a dr-local orders. A shorter checkpoint interval would have shrunk the reprocess window; a longer one would have grown it. Open at this step: step 5, Consumer catches up on dr; a new produce lands on the promoted region

The takeaway

Offset translation lets a consumer resume on the DR cluster after failover, but only as of the last checkpoint - so the checkpoint cadence sets the reprocess window (at-least-once across clusters), even when RPO is zero.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.2.4" title="Kafka Simulator: Offset translation across clusters" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/offset-translation-checkpoint/">MirrorMaker 2 offset translation for Kafka consumer failover</a></p>

Further reading

Apache Kafka documentation

More in Disaster recovery

All scenarios