Monedula
Kafka Simulator docs menu

Run a stretched cluster across data centers

Start a stretched free-play cluster, one Kafka cluster across three data centers, then lose a data center and see whether writes and elections survive.

How-to guides 5 of 10

3 min read

Updated 13 October 2026

Markdown

On this page

Steps

1. Start the sandbox

In the left rail of the playground, set cluster type to 3-DC stretched. On a narrow screen, pick it from the scenario menu in the top bar, under ✦ Free play. The sandbox starts with:

  • three data centers (DCs), dc-a, dc-b and dc-c, each holding one broker and one KRaft controller: broker-1 and c-a in dc-a, and so on;
  • the topic orders, with three partitions, replication factor 3 and min.insync.replicas 2, each partition led from a different DC;
  • the producer p1, with acks=all, and the consumer group cg1.

The Why tab ends with a DR posture card that reads “Stretched multi-DC · synchronous”: RPO “≈ 0”, RTO “seconds”.

2. Look at a data center

Select the dc-a band on the canvas. The Inspector opens on Data center · dc-a: whether it is reachable from the active controller, its brokers and its controllers, with the active one marked.

3. Lose one data center

Select kill · hard dc-a. broker-1 and c-a go down. The partition broker-1 led elects a new leader in another DC, and the two surviving controllers keep the quorum. The producer keeps writing. Select a partition of orders: its in-sync replicas list two brokers and its leader is outside dc-a.

4. Lose a second data center

Select the dc-b band and kill · hard dc-b. With one controller of three left, the quorum is lost and leader elections freeze. Partitions whose leader was in dc-a or dc-b stay without a leader, and the producer gets LeaderNotAvailable errors. The DR posture card adds “Quorum is lost right now - leadership is frozen until it returns.”

5. Recover

Select the dc-b band and restart dc-b. With two controllers of three back, the quorum returns, the leaderless partitions elect leaders again and the quorum note leaves the DR posture card. Restart dc-a the same way.

The 2.5-DC layout. Set cluster type to 2.5-DC stretched for the layout with a witness. It has two data DCs, dc-a and dc-b, each with three brokers: two synchronous replicas and one observer, an asynchronous replica that does not count toward the in-sync replicas. A third site, dc-c, labelled dc-c · witness on the canvas, holds only a controller, so the quorum keeps a majority when either data DC fails. The topic orders has four synchronous replicas and min.insync.replicas 3.

Kill the witness first: select the dc-c · witness band and kill · hard dc-c. Two controllers of three remain and writes go on. Restart it, then kill dc-a, which holds the partition’s leader and the active controller. Leadership moves to dc-b, the observer there is promoted into the in-sync replicas, and after a few retries the producer writes again.

The Failure tab has DC-level controls too: partition dc-a fences a DC off from the others until you select heal, kill dc-a destroys it, and kill KRaft · dc-a stops only its controller while its brokers stay up.