--- title: "Kafka Simulator v1.5 + v1.6 — stretched clusters, storage and ops" date: 2026-09-02T00:00:00.000Z author: "michal" excerpt: "Two packs at once. Free play completes its topology ladder with the 3-DC stretched and 2.5-DC witness clusters, and 19 new scenarios cover the log's lifecycle — retention, compaction, tiered storage — plus the KRaft control plane and the operations you run against a live cluster." --- Two packs ship together: **v1.5 — Storage & lifecycle** and **v1.6 — Ops/controller & quotas**. Between them the [Kafka Simulator](/kafka-simulator/) gains **19 new scenarios**, reaching **74 of 122**, and free play finishes the cluster-shape ladder it started in v1.3. We are leading with free play this time, because the two unlocks are the ones people have been asking for. ## Free play: the topology ladder is complete Since v1.3 the sandbox has been growing one cluster shape per pack: active/passive, then active/active. Both are **two clusters** joined by an asynchronous mirror. The two shapes landing now are the other kind: **one cluster stretched across datacenters**, where replication is synchronous and the ISR itself spans the WAN. The difference shows up the moment something fails: - In a **mirrored** pair, losing a region is a *failover*: you promote the other side and accept whatever the mirror had not yet copied. - In a **stretched** cluster, losing a DC is a *replication event*: the ISR shrinks, and whether you can still write depends entirely on `min.insync.replicas`. ### 3-DC stretched (new in v1.5) Three datacenters, one cluster. The default sandbox gives you a broker and a controller in each of `dc-a`, `dc-b` and `dc-c`, with brokers interleaved so round-robin placement spreads every partition's replicas across all three sites. Three partitions, so each DC opens holding a natural leader, so the layout shows balanced cross-DC leadership instead of one lonely partition. There is one **KRaft quorum distributed across the three sites**, not one per region. Kill a DC and you lose one voter out of three; the remaining two still form a majority and the cluster keeps making decisions. ### 2.5-DC stretched (new in v1.6) The last rung: two *data* datacenters plus a third **witness** site that holds a controller vote and no data at all. This is the Confluent MRC shape. Each data DC gets **2 sync replicas and 1 observer** — RF 4 across the sync set, with `min.insync.replicas` 3. Two things about the observers: - They **do not count toward the ISR**, so they do not hold up an `acks=all` write. A cross-WAN replica that gated every commit would be a latency disaster; an observer is how you keep a remote copy without paying for it on the write path. - When the ISR drops below `min.insync.replicas`, an observer can be **auto-promoted** into it to restore durability. The sandbox default promotes on an ISR drop and demotes again when the DC recovers; you can also have it stay promoted, or turn promotion off entirely and do it by hand. The witness pairs directly with this release's controller scenarios. A quorum wants an odd number of voters, and buying a third full datacenter to get one is expensive — so you buy half of one. Scenario 06.0.2 ("KRaft quorum") ends on exactly the failure the witness is there to prevent; now you can build both setups in the sandbox and compare. With this, **all five cluster shapes are live**: single DC, active/passive, active/active, 3-DC stretched and 2.5-DC stretched. Every remaining free-play unlock — the failure lab in v1.7, the Kafka CLI terminal in v1.8, governance in v1.9 — is a new capability rather than a new shape. Same caveat as in v1.3: the authored **DR curriculum** that narrates these topologies still lands in v1.8. The shapes arrive first on purpose, so there is somewhere to try the ideas before the scenarios explain them. If you want to read more about multi-region architectures in the meantime, SoftwareMill's [guide to Apache Kafka disaster recovery and multi-region architectures](https://softwaremill.com/guide-to-apache-kafka-disaster-recovery-and-multi-region-architectures/) is a thorough treatment of the trade-offs these shapes encode. Worth having open in another tab while you play with the sandbox. ## The scenarios ### v1.5 — Storage & lifecycle (9 scenarios) A Kafka log is not an infinite tape. This module covers what happens to records after they are written, in three groups. **Retention** — `retention.ms` and `retention.bytes` advancing the log start offset, and what a consumer gets when it asks for an offset that retention already deleted (`OFFSET_OUT_OF_RANGE`, and the reset that follows). **Compaction** — `cleanup.policy=compact` keeping the latest value per key while preserving offsets and leaving gaps; tombstones deleting a key, with `delete.retention.ms` as the grace window a consumer has to observe the deletion before it is purged; and `compact,delete` running both policies together. **Tiered storage** — offloading an aged log prefix to the remote tier, reading it back, and the difference between *local* retention and *total* retention. This is the group that most changes how you size a cluster: local retention stops being the thing that bounds how far back a consumer can read. ### v1.6 — the control plane (`kraft`, 4 scenarios) What the controller does, and what stops when it is gone: - **The controller's job** and the **KRaft quorum** — the controller keeps metadata consistent, not data. Lose the quorum's majority and there is no controller: leaders stay where they are and nothing new gets decided until a voter comes back. - **Controller failover and metadata catch-up** — a new active controller must reach the committed metadata offset before it can serve, which is why elections can freeze for a moment after a failover. - **ZooKeeper vs KRaft** — the same cluster under two control planes, side by side. ### v1.6 — operations (`operations`, 6 scenarios) What an operator changes on a live cluster, and what each change costs: - **Scale out** and the rebalance that follows; **scale in**, draining a broker before removing it. - **Partition reassignment by hand**, and the same reassignment **throttled under load** — where you watch a new replica join the ISR late because you capped its catch-up rate to protect live traffic. - **Live config changes**, and **client quotas** throttling a producer by delaying its responses rather than dropping data. ## What's next 74 of 122 scenarios, seven packs shipped, and a free-play sandbox that can now model every cluster shape the rest of the curriculum will reference. Next up is v1.7 and the failure lab: injecting broker kills and network partitions into any of those five shapes, on demand. Open the [simulator](/kafka-simulator/), start a 2.5-DC sandbox, and kill a data DC. Then watch an observer get promoted to keep you above `min.insync.replicas`.