Scenario 09.2.2 · Failover + RPO
Kafka DR failover with MirrorMaker 2: measuring RPO
Cut the mirror link, watch lag grow, fail over to the DR region - and measure the records lost (RPO).
Run this scenario in the playgroundTwo regions ready - primary serves orders, dr mirrors; consumer on primary. t+0.000s
Same active-passive setup as 09.2.1: primary serves orders and mm-1 mirrors it into dr's primary.orders. This time consumer group g1 (member c1) reads orders on primary, so a failover has to move a consumer as well as a producer.
Open at this step: step 1, Two regions ready - primary serves orders, dr mirrors; consumer on primaryMirror caught up (dr HW 3, lag 0); consumer following on primary. t+1.700s
p1 has written O1, O2 and O3 to primary (HW 3) and g1 has consumed and committed all three (committed = 3). MM2 copied them into dr's primary.orders at offsets 0, 1 and 2 (dr HW 3), so right now a failover would lose nothing.
Open at this step: step 2, Mirror caught up (dr HW 3, lag 0); consumer following on primaryLink cut - lag grows (src HW 5, dr 3, lag 2); consumer committed 5 on primary. t+2.900s
The regions were split at t=2000. p1 kept writing O4 and O5 to primary (HW 5) and g1 committed 5 there, but dr's primary.orders is still at 3. That lag of 2 records is the RPO you would pay on a failover now.
Open at this step: step 3, Link cut - lag grows (src HW 5, dr 3, lag 2); consumer committed 5 on primaryFailover to dr - RPO 2 records lost; consumer resumes on dr at clamped offset 3. t+3.100s
The failover at t=3000 promotes dr (promotedRegion = dr) and moves p1 there. g1 is re-pointed to primary.orders and resumes at 3, dr's high watermark: the simulator clamps its committed 5, where real MM2 would resume it from the last translated checkpoint, never past the mirrored 3. O4 and O5 exist only on the unreachable primary: RPO is 2 records.
Open at this step: step 4, Failover to dr - RPO 2 records lost; consumer resumes on dr at clamped offset 3Promoted: producer now writes dr, consumer reads the new record. t+3.400s
p1 produces O6 to dr's primary.orders, where it lands at offset 3 (HW 4), the offset O4 had on primary, and c1 reads it (g1 commits 4). Writing the mirror topic is a simulator shortcut: MM2 expects producers to write only local topics, so a real p1 would write a dr-local orders. Either way O4 and O5 are missing from dr's history.
Open at this step: step 5, Promoted: producer now writes dr, consumer reads the new recordWhat you start with
Builds on the active-passive mirror: after lag has grown on the inter-region link, we fail the application over to the DR cluster and count the records that never made it across. Those un-mirrored records are the recovery point objective (RPO) - the data lost to the outage.
Step by step
- Two regions ready - primary serves orders, dr mirrors; consumer on primary t+0.000s Same active-passive setup as 09.2.1: primary serves orders and mm-1 mirrors it into dr's primary.orders. This time consumer group g1 (member c1) reads orders on primary, so a failover has to move a consumer as well as a producer. Open at this step: step 1, Two regions ready - primary serves orders, dr mirrors; consumer on primary
- Mirror caught up (dr HW 3, lag 0); consumer following on primary t+1.700s p1 has written O1, O2 and O3 to primary (HW 3) and g1 has consumed and committed all three (committed = 3). MM2 copied them into dr's primary.orders at offsets 0, 1 and 2 (dr HW 3), so right now a failover would lose nothing. Open at this step: step 2, Mirror caught up (dr HW 3, lag 0); consumer following on primary
- Link cut - lag grows (src HW 5, dr 3, lag 2); consumer committed 5 on primary t+2.900s The regions were split at t=2000. p1 kept writing O4 and O5 to primary (HW 5) and g1 committed 5 there, but dr's primary.orders is still at 3. That lag of 2 records is the RPO you would pay on a failover now. Open at this step: step 3, Link cut - lag grows (src HW 5, dr 3, lag 2); consumer committed 5 on primary
- Failover to dr - RPO 2 records lost; consumer resumes on dr at clamped offset 3 t+3.100s The failover at t=3000 promotes dr (promotedRegion = dr) and moves p1 there. g1 is re-pointed to primary.orders and resumes at 3, dr's high watermark: the simulator clamps its committed 5, where real MM2 would resume it from the last translated checkpoint, never past the mirrored 3. O4 and O5 exist only on the unreachable primary: RPO is 2 records. Open at this step: step 4, Failover to dr - RPO 2 records lost; consumer resumes on dr at clamped offset 3
- Promoted: producer now writes dr, consumer reads the new record t+3.400s p1 produces O6 to dr's primary.orders, where it lands at offset 3 (HW 4), the offset O4 had on primary, and c1 reads it (g1 commits 4). Writing the mirror topic is a simulator shortcut: MM2 expects producers to write only local topics, so a real p1 would write a dr-local orders. Either way O4 and O5 are missing from dr's history. Open at this step: step 5, Promoted: producer now writes dr, consumer reads the new record
The takeaway
RPO is the measurable cost of async DR: any records produced after mirroring fell behind are lost on failover. Smaller mirror lag means a smaller RPO, but with async replication it is never zero.
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.2.2" title="Kafka Simulator: Failover + RPO" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/dr-failover/">Kafka DR failover with MirrorMaker 2: measuring RPO</a></p>Further reading
More in Disaster recovery
- Kafka stretched cluster across two data centers
- 2.5-DC Kafka stretched cluster with a witness site
- 3-DC Kafka cluster: the KRaft quorum survives losing a site
- Kafka in two DCs without a witness: KRaft quorum is lost
- Kafka data center failure: leader failover to the other DC
- Kafka network split inside one DC: ISR drops to the floor
- Kafka MirrorMaker 2 active-passive DR and replication lag
- Active-active Kafka with MirrorMaker 2 and region topics
- MirrorMaker 2 offset translation for Kafka consumer failover
- Kafka failback after a MirrorMaker 2 DR failover
- Active-active Kafka DR drill: move clients to the survivor
- Confluent MRC observers: Kafka replicas outside the ISR
- Kafka observer lag in Confluent Multi-Region Clusters
- Kafka observer promotion in Confluent Multi-Region Clusters
- Confluent MRC observer promotion: losing the Kafka leader DC
- Kafka replica placement: durability vs latency across DCs
- Confluent Cluster Linking: offset-preserving Kafka mirroring
- Kafka mirror topics (Confluent): read-only until promoted
- Cluster Linking vs MirrorMaker 2 for Kafka replication
- Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2
- Kafka failover and failback with Confluent Cluster Linking