Scenario 09.2.5 · Failback after the primary recovers
Kafka failback after a MirrorMaker 2 DR failover
Fail over to DR, take writes there during the outage, then fail back - the engine reverse-re-syncs DR→primary so the recovered primary catches up with no data loss and forward mirroring resumes.
Run this scenario in the playgroundTwo regions ready - primary serves orders, MM2 mirrors to dr. t+0.000s
primary holds the orders leader; the mm-1 link copies committed records into dr's primary.orders. Producer p1 and consumer g1 are on the primary.
Open at this step: step 1, Two regions ready - primary serves orders, MM2 mirrors to drSteady state: O1..O3 produced, mirrored, and drained. t+0.700s
g1 has consumed offsets 0,1,2 on primary (committed = 3). MM2 has copied all three into dr's primary.orders (remote HW = 3), so RPO = 0.
Open at this step: step 2, Steady state: O1..O3 produced, mirrored, and drainedFailover to dr - consumer resumes on primary.orders, producers redirected. t+1.100s
promotedRegion = dr. g1 is re-pointed onto dr's primary.orders at the translated offset (3); p1 now writes to dr's primary.orders, a simulator shortcut: MM2 expects producers to write only local topics, so a real p1 would write a dr-local orders. The primary is treated as down.
Open at this step: step 3, Failover to dr - consumer resumes on primary.orders, producers redirectedOutage-window writes land on dr (O4, O5). t+1.600s
p1 produces O4,O5 to dr's primary.orders (HW = 5), the simulator shortcut again (a real p1 would write a dr-local orders); g1 drains them (committed = 5). The primary's orders log is unchanged at 3 - these writes exist only on dr.
Open at this step: step 4, Outage-window writes land on dr (O4, O5)Failback to primary - reverse re-sync pulls the dr writes back. t+2.100s
The same failover action, now targeting primary, reverse-re-syncs dr→primary: orders catches up to the dr HW (5), g1 is re-pointed back to orders with inverted offsets, the forward mirror is re-armed, producers return to primary and promotedRegion is cleared. No data lost.
Open at this step: step 5, Failback to primary - reverse re-sync pulls the dr writes backRestored primary takes a new write - forward mirroring resumed. t+2.600s
p1 produces O6 to the recovered primary's orders (HW = 6); MM2 mirrors it onward to dr (remote HW = 6) and g1 keeps draining (committed = 6). Failback complete and consistent.
Open at this step: step 6, Restored primary takes a new write - forward mirroring resumedWhat you start with
The companion to failover: after running on DR during an outage - including new writes taken there - the original primary comes back and we fail back to it. Rather than a cold second failover, the engine reverse-re-syncs DR→primary so the outage-window writes flow back, the consumer re-points to the original topic with its offsets translated, and forward primary→DR mirroring re-arms.
Step by step
- Two regions ready - primary serves orders, MM2 mirrors to dr t+0.000s primary holds the orders leader; the mm-1 link copies committed records into dr's primary.orders. Producer p1 and consumer g1 are on the primary. Open at this step: step 1, Two regions ready - primary serves orders, MM2 mirrors to dr
- Steady state: O1..O3 produced, mirrored, and drained t+0.700s g1 has consumed offsets 0,1,2 on primary (committed = 3). MM2 has copied all three into dr's primary.orders (remote HW = 3), so RPO = 0. Open at this step: step 2, Steady state: O1..O3 produced, mirrored, and drained
- Failover to dr - consumer resumes on primary.orders, producers redirected t+1.100s promotedRegion = dr. g1 is re-pointed onto dr's primary.orders at the translated offset (3); p1 now writes to dr's primary.orders, a simulator shortcut: MM2 expects producers to write only local topics, so a real p1 would write a dr-local orders. The primary is treated as down. Open at this step: step 3, Failover to dr - consumer resumes on primary.orders, producers redirected
- Outage-window writes land on dr (O4, O5) t+1.600s p1 produces O4,O5 to dr's primary.orders (HW = 5), the simulator shortcut again (a real p1 would write a dr-local orders); g1 drains them (committed = 5). The primary's orders log is unchanged at 3 - these writes exist only on dr. Open at this step: step 4, Outage-window writes land on dr (O4, O5)
- Failback to primary - reverse re-sync pulls the dr writes back t+2.100s The same failover action, now targeting primary, reverse-re-syncs dr→primary: orders catches up to the dr HW (5), g1 is re-pointed back to orders with inverted offsets, the forward mirror is re-armed, producers return to primary and promotedRegion is cleared. No data lost. Open at this step: step 5, Failback to primary - reverse re-sync pulls the dr writes back
- Restored primary takes a new write - forward mirroring resumed t+2.600s p1 produces O6 to the recovered primary's orders (HW = 6); MM2 mirrors it onward to dr (remote HW = 6) and g1 keeps draining (committed = 6). Failback complete and consistent. Open at this step: step 6, Restored primary takes a new write - forward mirroring resumed
The takeaway
In this simulator failback is a reverse re-sync, not a cold restart: the writes taken on DR during the outage are pulled back onto the recovered primary with no data loss, and the consumer continues without a gap. Real MirrorMaker 2 has no failback action: a dr-to-primary flow skips primary.orders as a cycle, so outage writes belong in a dr-local topic that mirrors back as dr.orders.
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.2.5" title="Kafka Simulator: Failback after the primary recovers" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/failback-after-primary-recovers/">Kafka failback after a MirrorMaker 2 DR failover</a></p>Further reading
More in Disaster recovery
- Kafka stretched cluster across two data centers
- 2.5-DC Kafka stretched cluster with a witness site
- 3-DC Kafka cluster: the KRaft quorum survives losing a site
- Kafka in two DCs without a witness: KRaft quorum is lost
- Kafka data center failure: leader failover to the other DC
- Kafka network split inside one DC: ISR drops to the floor
- Kafka MirrorMaker 2 active-passive DR and replication lag
- Kafka DR failover with MirrorMaker 2: measuring RPO
- Active-active Kafka with MirrorMaker 2 and region topics
- MirrorMaker 2 offset translation for Kafka consumer failover
- Active-active Kafka DR drill: move clients to the survivor
- Confluent MRC observers: Kafka replicas outside the ISR
- Kafka observer lag in Confluent Multi-Region Clusters
- Kafka observer promotion in Confluent Multi-Region Clusters
- Confluent MRC observer promotion: losing the Kafka leader DC
- Kafka replica placement: durability vs latency across DCs
- Confluent Cluster Linking: offset-preserving Kafka mirroring
- Kafka mirror topics (Confluent): read-only until promoted
- Cluster Linking vs MirrorMaker 2 for Kafka replication
- Kafka consumer offset sync: Cluster Linking vs MirrorMaker 2
- Kafka failover and failback with Confluent Cluster Linking