--- title: "Active-active Kafka DR drill: move clients to the survivor" description: "Lose the west Kafka region, move pw and cg-west to east and keep writing. Ow2 never mirrored, so it is the RPO; when west returns, MM2 delivers it late." url: https://monedula.dev/kafka-simulator/scenarios/active-active-dr-drill/ --- # Active-active Kafka DR drill: move clients to the survivor Scenario 09.2.6 · Active-active DR drill. Disaster recovery · Advanced · about 5 min · Apache Kafka 4.3. Lose west, move its producer and consumer group to east, keep writing, then restart west and move them back. Nothing is promoted; the unmirrored tail is the price. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6 ## What you start with Active-active has no standby to promote: both regions already take writes. Failing over means moving the lost region’s clients to the survivor. We lose west with one record still inside the mirror lag, redirect pw and cg-west to east, keep producing, then restart west and move its clients back home. ## Step by step 1. **Two active regions - each owns a topic and mirrors it to the other** (t+0.000s): The 09.2.3 topology: west owns west.orders, east owns east.orders, and mm-w2e / mm-e2w copy each one into the other region under the same name. pw writes west.orders in west, pe writes east.orders in east, and each region's group reads both topics locally. There is no standby, so there is nothing to promote when a region dies. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=0) 2. **Both regions wrote and mirrored one record (lag 0 each way)** (t+1.700s): Ow1 and Oe1 are in both regions: each local topic is at HW 1 and each mirrored copy caught up at t=1500. cg-west and cg-east have each committed 1 on both topics. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=1) 3. **pw writes Ow2 in west - not mirrored yet (mm-w2e lag 1)** (t+2.300s): Ow2 lands in west.orders (HW 2) and cg-west reads it, but the mirror copies a record only after it has aged 600 ms, so east's copy of west.orders is still at HW 1. If west dies now, Ow2 is the RPO: the one record east does not have. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=2) 4. **west lost - brokers and KRaft voters down, Ow2 stranded** (t+2.500s): kill_dc west at t=2400 took broker-1, broker-2 and the three west voters down. west.orders has no leader, so pw can no longer write and cg-west can no longer read. east keeps serving: pe writes east.orders and cg-east reads in east. mm-w2e is frozen at offset 1 with Ow2 still in west. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=3) 5. **west clients moved to east - pw writes east.orders, cg-west resumes on the mirror** (t+2.700s): redirect_clients west -> east at t=2600. pw reconnects to east as a new producer session (fresh PID) and, because west.orders is a read-only mirror in east, writes east's own topic east.orders. cg-west re-pins to east: on the mirror of west.orders it resumes at the translated offset clamped to the mirror HW, 1, because Ow2 never reached east; on east.orders it keeps offset 1. Nothing was promoted. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=4) 6. **Both producers write east.orders in east (HW 3)** (t+3.200s): pw's Ow3 and pe's Oe2 land in east.orders (HW 3), and both groups read them in east. cg-west committed 3 on east.orders. west is still down, so its copy of east.orders stays at HW 1. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=5) 7. **west restarted - its disk still had Ow2, so the mirror delivers it late** (t+4.200s): broker-1, broker-2 and the west voters restarted at t=3800 and re-elected leaders. Both mirrors resumed: mm-e2w copied Ow3 and Oe2 into west (HW 3) and mm-w2e copied Ow2 into east (HW 2). cg-west, still on east, read Ow2 there although it had already read it in west before the loss: a duplicate. Had west not come back, Ow2 would be gone. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=6) 8. **west clients moved back home** (t+5.100s): restore_clients west at t=5000. pw reconnects to west as a new session and writes west.orders again. cg-west re-pins to west and carries its positions back: west.orders at 2 and east.orders at 3, so it neither skips nor re-reads anything. pe and cg-east never moved. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=7) 9. **Back to normal - every copy of both topics at HW 3 (lag 0)** (t+6.200s): pw's Ow4 landed in west.orders (HW 3) and mm-w2e copied it to east at t=6000. Both groups have committed 3 on both topics in their home regions. The drill cost one record of exposure (Ow2) and one duplicate read, and no promotion or failback of a cluster. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=09.2.6&t=8) ## The takeaway An active-active failover is a client move, not a promotion. Producers reconnect to the survivor as new sessions and write its own topic, consumer groups resume on the survivor’s copies at the translated offset, and whatever the lost region never mirrored is the RPO. If the region comes back with its disks, the mirror delivers that tail late; if it does not, those records are gone. ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/operations/geo-replication-cross-cluster-data-mirroring/)