Monedula

Scenario 10.1.5 · Durability vs latency placement

Kafka replica placement: durability vs latency across DCs

Disaster recovery · Advanced · about 4 min · Apache Kafka 4.3 · Confluent Platform feature

Two topics on the same 2-DC cluster make the MRC placement trade-off concrete. orders-durable keeps a synchronous replica in EACH DC (min.ISR 2): every acks=all write waits for the remote DC to confirm - higher commit latency, but a DC loss fails over with the committed data intact. orders-fast keeps one local sync replica (min.ISR 1) and an async observer in the other DC: writes commit locally and instantly - lower latency, but when its DC dies the partition is stranded leaderless and the last local write is at risk. Same cluster, opposite guarantees, chosen entirely by replica placement.

Run this scenario in the playground

DC-A lost - durable fails over with data intact (read-only); fast is stranded leaderless. t+3.400s

Losing DC-A: orders-durable re-elects its DC-B sync replica (broker-2) with all committed data (HW 2, no loss) - though with ISR 1 < min.ISR 2 it is temporarily read-only. orders-fast has no surviving sync replica, so it is left LEADERLESS - its async observer (broker-2, only at LEO 1) can't be promoted without a leader, and the last local write (offset 1) is stranded on the unreachable DC-A leader.

Open at this step: step 5, DC-A lost - durable fails over with data intact (read-only); fast is stranded leaderless

What you start with

Two topics on one 2-DC MRC cluster make the placement trade-off concrete. orders-durable keeps a synchronous replica in each DC (min.ISR 2), so every acks=all write waits for the remote DC to confirm; orders-fast keeps one local sync replica (min.ISR 1) plus a remote async observer, so writes commit instantly.

Step by step

  1. Two placements on one cluster - orders-durable (sync in both DCs) vs orders-fast (local sync + remote observer) t+0.000s orders-durable has a synchronous replica in each DC (min.ISR 2). orders-fast has one sync replica in DC-A (min.ISR 1) plus an async observer in DC-B. Both lead in DC-A; only the second copy's placement differs. Open at this step: step 1, Two placements on one cluster - orders-durable (sync in both DCs) vs orders-fast (local sync + remote observer)
  2. Durable write D1 waits for the remote DC - committed at ack latency 200ms t+0.900s p-dur's acks=all write only commits once DC-B's synchronous replica confirms, so the high-watermark advances after the cross-DC round trip (replicaFetchMs 200) and the producer's ack latency is 200ms. The price of cross-DC durability is paid on every write. Open at this step: step 2, Durable write D1 waits for the remote DC - committed at ack latency 200ms
  3. Fast write F1 commits locally - ack latency 0ms, observer trails t+1.300s p-fast's acks=all write commits on the local DC-A leader alone (min.ISR 1), so its ack latency is ~0ms - far quicker than the durable topic. The DC-B observer (broker-2) hasn't fetched it yet (LEO 0 < HW 1): the second copy is asynchronous. Open at this step: step 3, Fast write F1 commits locally - ack latency 0ms, observer trails
  4. Steady state - durable mirrored synchronously (HW 2 both DCs), fast leads locally with the observer lagging t+2.400s After two writes each: orders-durable has both records on both DCs (LEO 2/2) at a 200ms commit cost; orders-fast has both on its DC-A leader (HW 2) at ~0ms cost, with its observer one behind (broker-2 LEO 1). Low latency now means the remote copy is not yet current. Open at this step: step 4, Steady state - durable mirrored synchronously (HW 2 both DCs), fast leads locally with the observer lagging
  5. DC-A lost - durable fails over with data intact (read-only); fast is stranded leaderless t+3.400s Losing DC-A: orders-durable re-elects its DC-B sync replica (broker-2) with all committed data (HW 2, no loss) - though with ISR 1 < min.ISR 2 it is temporarily read-only. orders-fast has no surviving sync replica, so it is left LEADERLESS - its async observer (broker-2, only at LEO 1) can't be promoted without a leader, and the last local write (offset 1) is stranded on the unreachable DC-A leader. Open at this step: step 5, DC-A lost - durable fails over with data intact (read-only); fast is stranded leaderless
  6. Healed - durable fully recovered and writable again; fast re-elects broker-1 t+5.000s Reconnecting DC-A: orders-durable's original replica rejoins the ISR (back to 2 ≥ min.ISR), so the topic is durable AND writable again - it rode out the whole outage. orders-fast was unavailable for the whole outage and only recovers because broker-1, its last in-sync replica, came back: the controller re-elects it cleanly (HW 2, nothing lost). Had DC-A never returned, only an unclean election of the observer (LEO 1) could bring it back, losing the last write. Latency was cheaper, durability was not. Open at this step: step 6, Healed - durable fully recovered and writable again; fast re-elects broker-1

The takeaway

Replica placement alone sets the durability-vs-latency guarantee: cross-DC synchronous commit survives a region loss with no data loss but adds latency, while local-only commit is fast but risks the last writes when its DC dies.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.1.5" title="Kafka Simulator: Durability vs latency placement" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/durability-vs-latency/">Kafka replica placement: durability vs latency across DCs</a></p>

Further reading

Confluent documentation

More in Disaster recovery

All scenarios