Monedula

Scenario 09.1.6 · Intra-DC partition split

Kafka network split inside one DC: ISR drops to the floor

Disaster recovery · Advanced · about 3 min · Apache Kafka 4.3

A link fails INSIDE one DC, splitting that DC’s own brokers across the partition - the ISR sheds the cut replica to the min.ISR floor while the other DC stays whole.

Run this scenario in the playground

Surviving replicas keep committing - write acked on the shrunk ISR of 3. t+12.000s

Open at this step: step 4, Surviving replicas keep committing - write acked on the shrunk ISR of 3

What you start with

Reusing the 2.5-DC MRC layout, this time the network failure lands inside one data centre: a single broker is cut off from its own DC peers and everything else. The controller quorum and the leader stay together in the majority, so only the isolated broker drops out of the ISR. We watch the ISR fall to exactly the min.insync.replicas floor while the other DC stays whole, then heal.

Step by step

  1. Stretched cluster ready - DC-A & DC-B each 2 sync + 1 observer, witness DC-C t+0.000s Open at this step: step 1, Stretched cluster ready - DC-A & DC-B each 2 sync + 1 observer, witness DC-C
  2. Record committed across both DCs (ISR = 4 sync replicas ≥ min.ISR 3) t+5.000s Open at this step: step 2, Record committed across both DCs (ISR = 4 sync replicas ≥ min.ISR 3)
  3. Intra-DC link cut - DC-A splits internally; broker-2 leaves the ISR (now 3 = min.ISR) t+8.000s A network partition lands INSIDE DC-A: broker-2 is cut off from its own DC peers (broker-1, broker-3) and the rest of the cluster, while DC-B stays whole. The fence renders inside the DC-A band, not between DCs. Every controller voter and the leader stay together, so the quorum and leadership are untouched - the active controller simply can no longer reach broker-2, which drops out of the ISR. The ISR falls to exactly the min.ISR floor of 3, so no observer is promoted. Open at this step: step 3, Intra-DC link cut - DC-A splits internally; broker-2 leaves the ISR (now 3 = min.ISR)
  4. Surviving replicas keep committing - write acked on the shrunk ISR of 3 t+12.000s Open at this step: step 4, Surviving replicas keep committing - write acked on the shrunk ISR of 3
  5. Healed - broker-2 catches up and rejoins the ISR (back to 4 sync replicas) t+14.000s Open at this step: step 5, Healed - broker-2 catches up and rejoins the ISR (back to 4 sync replicas)

The takeaway

A partition need not fall on a DC boundary - an intra-DC link failure simply sheds the unreachable replica from the ISR. As long as the survivors still meet min.insync.replicas, writes continue at the floor with no observer promotion.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=09.1.6" title="Kafka Simulator: Intra-DC partition split" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/intra-dc-partition-split/">Kafka network split inside one DC: ISR drops to the floor</a></p>

Further reading

Apache Kafka documentation

More in Disaster recovery

All scenarios