Scenario 09.2.6 · Active-active DR drill

Active-active Kafka DR drill: move clients to the survivor

Disaster recovery · Advanced · about 5 min · Apache Kafka 4.3

Lose west, move its producer and consumer group to east, keep writing, then restart west and move them back. Nothing is promoted; the unmirrored tail is the price.

Run this scenario in the playground

west clients moved to east - pw writes east.orders, cg-west resumes on the mirror. t+2.700s

redirect_clients west -> east at t=2600. pw reconnects to east as a new producer session (fresh PID) and, because west.orders is a read-only mirror in east, writes east's own topic east.orders. cg-west re-pins to east: on the mirror of west.orders it resumes at the translated offset clamped to the mirror HW, 1, because Ow2 never reached east; on east.orders it keeps offset 1. Nothing was promoted.

Open at this step: step 5, west clients moved to east - pw writes east.orders, cg-west resumes on the mirror

What you start with

Active-active has no standby to promote: both regions already take writes. Failing over means moving the lost region’s clients to the survivor. We lose west with one record still inside the mirror lag, redirect pw and cg-west to east, keep producing, then restart west and move its clients back home.

Step by step

Two active regions - each owns a topic and mirrors it to the other t+0.000s The 09.2.3 topology: west owns west.orders, east owns east.orders, and mm-w2e / mm-e2w copy each one into the other region under the same name. pw writes west.orders in west, pe writes east.orders in east, and each region's group reads both topics locally. There is no standby, so there is nothing to promote when a region dies. Open at this step: step 1, Two active regions - each owns a topic and mirrors it to the other
Both regions wrote and mirrored one record (lag 0 each way) t+1.700s Ow1 and Oe1 are in both regions: each local topic is at HW 1 and each mirrored copy caught up at t=1500. cg-west and cg-east have each committed 1 on both topics. Open at this step: step 2, Both regions wrote and mirrored one record (lag 0 each way)
pw writes Ow2 in west - not mirrored yet (mm-w2e lag 1) t+2.300s Ow2 lands in west.orders (HW 2) and cg-west reads it, but the mirror copies a record only after it has aged 600 ms, so east's copy of west.orders is still at HW 1. If west dies now, Ow2 is the RPO: the one record east does not have. Open at this step: step 3, pw writes Ow2 in west - not mirrored yet (mm-w2e lag 1)
west lost - brokers and KRaft voters down, Ow2 stranded t+2.500s kill_dc west at t=2400 took broker-1, broker-2 and the three west voters down. west.orders has no leader, so pw can no longer write and cg-west can no longer read. east keeps serving: pe writes east.orders and cg-east reads in east. mm-w2e is frozen at offset 1 with Ow2 still in west. Open at this step: step 4, west lost - brokers and KRaft voters down, Ow2 stranded
west clients moved to east - pw writes east.orders, cg-west resumes on the mirror t+2.700s redirect_clients west -> east at t=2600. pw reconnects to east as a new producer session (fresh PID) and, because west.orders is a read-only mirror in east, writes east's own topic east.orders. cg-west re-pins to east: on the mirror of west.orders it resumes at the translated offset clamped to the mirror HW, 1, because Ow2 never reached east; on east.orders it keeps offset 1. Nothing was promoted. Open at this step: step 5, west clients moved to east - pw writes east.orders, cg-west resumes on the mirror
Both producers write east.orders in east (HW 3) t+3.200s pw's Ow3 and pe's Oe2 land in east.orders (HW 3), and both groups read them in east. cg-west committed 3 on east.orders. west is still down, so its copy of east.orders stays at HW 1. Open at this step: step 6, Both producers write east.orders in east (HW 3)
west restarted - its disk still had Ow2, so the mirror delivers it late t+4.200s broker-1, broker-2 and the west voters restarted at t=3800 and re-elected leaders. Both mirrors resumed: mm-e2w copied Ow3 and Oe2 into west (HW 3) and mm-w2e copied Ow2 into east (HW 2). cg-west, still on east, read Ow2 there although it had already read it in west before the loss: a duplicate. Had west not come back, Ow2 would be gone. Open at this step: step 7, west restarted - its disk still had Ow2, so the mirror delivers it late
west clients moved back home t+5.100s restore_clients west at t=5000. pw reconnects to west as a new session and writes west.orders again. cg-west re-pins to west and carries its positions back: west.orders at 2 and east.orders at 3, so it neither skips nor re-reads anything. pe and cg-east never moved. Open at this step: step 8, west clients moved back home
Back to normal - every copy of both topics at HW 3 (lag 0) t+6.200s pw's Ow4 landed in west.orders (HW 3) and mm-w2e copied it to east at t=6000. Both groups have committed 3 on both topics in their home regions. The drill cost one record of exposure (Ow2) and one duplicate read, and no promotion or failback of a cluster. Open at this step: step 9, Back to normal - every copy of both topics at HW 3 (lag 0)

The takeaway

An active-active failover is a client move, not a promotion. Producers reconnect to the survivor as new sessions and write its own topic, consumer groups resume on the survivor’s copies at the translated offset, and whatever the lost region never mirrored is the RPO. If the region comes back with its disks, the mirror delivers that tail late; if it does not, those records are gone.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.2.6" title="Kafka Simulator: Active-active DR drill" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/active-active-dr-drill/">Active-active Kafka DR drill: move clients to the survivor</a></p>

More in Disaster recovery

All scenarios