v2.0 is the last pack of the curriculum. It adds four modules - schema-registry, authorization, share-groups and capstones - plus three diskless topics scenarios in the storage module, an active-active DR drill in the dr module and rack-aware partitioning in the producer module, 24 new scenarios in all. It unlocks the governance and diskless tiers in free play, and completes the Kafka Simulator curriculum at 136 of 136.

It is also the release where the playground around those scenarios was rebuilt. Since v1.9 almost every part of it changed: the canvas has three views, the timeline became something you can read, the inspectors answer questions instead of listing fields, and the terminal no longer covers the canvas.

What’s in v2.0

Schema Registry (schema-registry, 3 scenarios). Subjects and versions, what backward, forward and full compatibility check, and a breaking change rejected before it is registered.

Authorization (authorization, 2 scenarios). Access control on the data path - how an ACL, or an RBAC role binding, allows or denies a produce when the authorizer denies by default.

Both come into free play with the new governance tier: schema registry, ACL and RBAC controls, and kafka-acls in the terminal, so you can rebuild and break by hand anything the scenarios show.

Share groups (share-groups, 9 scenarios). Kafka’s queue semantics, introduced in KIP-932:

Share groups change how Kafka is consumed, and seeing them next to classic consumer groups shows what is different.

Capstones (capstones, 5 scenarios). End-to-end scenarios that combine durability, transactions, storage, multi-DC and governance, with more than one thing going on at once: an exactly-once payments pipeline that survives a region loss, a zero-data-loss regional failover, a zero-downtime migration with Cluster Linking, a multi-DC active-active backbone, and a pipeline tuned for throughput and latency at the same time.

Disaster recovery (dr, 1 more scenario). An active-active DR drill: lose the west region, move its producer and consumer group to east and keep writing, then restart west and move them back. Nothing is promoted, and the records west never mirrored are the price.

Producer (producer, 1 more scenario). Rack-aware partitioning, new in Kafka 4.4: a producer keeps its records without a key in its own availability zone, until that zone loses its leaders. More on it under Apache Kafka 4.4 below.

Diskless topics (KIP-1150 direction, KIP-1163/1164 draft)

A diskless topic keeps its partition data in object storage instead of on broker disks. A broker buffers records from many partitions and uploads them as one WAL object; a diskless coordinator assigns the offsets when it commits that object, and the produce is acked at the commit. Replicas build their log by reading the object back from storage rather than fetching from the leader, and producers and consumers use a replica in their own availability zone.

The status: KIP-1150 is accepted as the direction, while KIP-1163 and KIP-1164, which define how it works, are still Under Discussion (KAFKA-19161 tracks the implementation). We model the current draft and mark it “KIP draft” throughout the simulator. Names and behaviour may change before any of it ships in Kafka.

Three scenarios (storage, 05.4.1-05.4.3):

  • Diskless: the write path - two producers and two partitions share one WAL object, no offset exists until the coordinator commits it, and the replicas read the data back from object storage.
  • No replication across AZs - the same 3 records go to a classic and a diskless topic across three AZs; the Cross-AZ traffic card ends at classic 7 and diskless 0 hops.
  • The price is latency - the same acks=all produce is acked in 300 ms on a classic topic and 1300 ms on a diskless one, while a burst that fills the buffer is acked in 800 ms. The timings are scaled up so each step is visible.

In free play: a Diskless 3 AZ preset, a per-topic diskless toggle and sliders for the diskless timing. The canvas gets an object storage lane on its own time axis, WAL trays under each broker and a coordinator badge, and the metrics panel a Cross-AZ traffic card. In the terminal, kafka-topics --create --config diskless.enable=true creates a diskless topic and kafka-diskless-metadata (itself a draft tool, not in Apache Kafka yet) shows per-partition offsets and lists the WAL objects in storage.

The trade-off in short: less cross-AZ replication traffic, slower produces - the draft targets a produce latency around 500 ms at P50 and 1-2 s at P99. Classic Kafka can already read from a same-AZ follower (KIP-392), so what diskless adds is no broker-to-broker replication and a produce that does not have to reach the leader.

Apache Kafka 4.4

The model now follows Apache Kafka 4.4, released this week. We compared 4.3.0 with 4.4.0 and found no default, CLI output or error message the simulator relies on that changed, so everything in the scenarios behaves as before.

v2.0 models one 4.4 addition: rack-aware producer partitioning (KIP-1123). With partitioner.rack.aware=true and a client.rack, the default partitioner sends records without a key to partitions whose leader is in the producer’s own availability zone, and to any partition when that zone has no leader, which keeps produce traffic inside the zone. The new scenario Rack-aware partitioning shows both cases. In free play the producer inspector has the switch, and the producers a sandbox starts with can pick their client.rack. Without a client.rack the producer refuses the setting, as Kafka does.

Share-group dead-letter queues (KIP-1191), another 4.4 addition, are not modeled yet.

A rebuilt playground

Three ways to see a cluster

The canvas now has three views, switched from its toolbar:

  • Brokers is the familiar picture: producers on the left, the cluster in the middle, consumer and share groups on the right.
  • Lanes draws one row per partition on a shared offset axis: the replicas under one another, leader first, the high watermark as one line through all of them. A lagging follower has a visibly shorter log, a stopped broker’s log stands still, and every consumer sits at its read position with its committed offset and lag. You see at a glance where everyone is in the log.
  • Matrix is for clusters too big to draw: a table with topics down and brokers across, and four lenses - Failures, Balance, Lag and Traffic - that colour and sort it worst first. Hovering a broker shows what the cluster would lose, topic by topic, before you stop it. A large free-play cluster opens in Matrix on its own.

The canvas shows what changed

State changes now play on the element that changed instead of a generic pulse: high-watermark markers slide to their new offset, the leader’s crown moves from the old leader to the new one, and a follower’s ISR tag takes the colour of the change. A produced record lands on its own log cell, a busy canvas draws records as short trails, and a stopped broker stays on the canvas, faded, so you can see what used to be there. On multi-region canvases the mirror link shows its RPO, each DC band says what losing it would cost, a network partition shows which side keeps serving, the KRaft strip counts its majority, and a share group’s lane shows each record’s lock timer and delivery count.

A timeline you can read

On the desktop the bottom dock is now a one-line rail: the frames sit at their real sim time, so a burst reads as a cluster and a quiet stretch as a gap, and you scrub by dragging along it. Open its event lanes and every event of the run is laid out on produce, replicate, brokers and consume lanes, failures in red, with each frame’s main event marked.

Inspectors that answer questions

The right panel now starts from the questions you ask. A partition reads as a ruler of its offsets, a broker as what you lose when it stops, a topic as a lane per partition, a producer as its send path, and a consumer group as lanes or a partition grid. Why lists what broke and why, Metrics are grouped by role, and some inspectors show what will happen next, not only the current state. The multi-region views get a DR card that shows each region’s log and the unmirrored tail as RPO.

The canvas makes room for the terminal

The Kafka CLI terminal still docks under the canvas, but the canvas now makes room for it instead of disappearing behind it - so kafka-server-stop and broker-1 going down are on screen together, and the terminal has the canvas’s full width for what you type and what the tools print. On a wide screen, dock side in its title bar moves it into a column beside the canvas instead. The output keeps its columns, and the host menu is an ssh table of every broker and controller. The broker config files match Kafka 4.4, and kafka-reassign-partitions --generate plus draining a broker before removing it work the way they do on a real cluster.

Getting started, without a tour

The 13-step modal tour is gone. In its place is a Getting started card in the corner of the canvas: the same topics as a checklist you work through in any order, each with a “Show me”. Help is now a reference you look things up in, a scripted scenario can be forked into a sandbox with Undo, scenarios suggest their own “Try it yourself” moves and what to do next, and a shared link now replays how it got to the state it shows.

A new look, and a pass on accessibility

The whole site, playground included, has a new look based on the jackdaw in our name: dark ink, a blue sheen and grey, with every text colour pair at WCAG AA. Along with it came skip links and named regions, a one-tab-stop scenario rail, AA contrast for inactive regions, press feedback and larger touch targets, and motion that respects prefers-reduced-motion by keeping fades and dropping movement. The phone layout got its own pass, down to a test that every label still fits.

Outside the playground

The playground itself is not indexable, so search engines never saw its scenario text. Now every scenario has its own page under /kafka-simulator/scenarios/: its intro and conclusion, every step with an “Open at this step” link, a real canvas frame you can step through, and the Kafka configs it touches. With v2.0 that is 136 pages.

Next to them are troubleshooting pages for Kafka errors and symptoms - what the problem means, the likely causes, the fixes and what not to do, whether the client retries it - each linked to the scenarios where it happens, opened at the moment it does. The authorization page joins them with this release. The changelog now reads as release notes on a timeline.

One more thing

We hid something in this release. We will leave finding it to you.

The end of the roadmap

v2.0 finishes what started at v1.0: eleven biweekly packs, 136 scenarios, from a single broker’s anatomy to queues and full-platform capstones. Everything runs in the browser, deterministically, on Apache Kafka 4.4 semantics, with no backend and no telemetry about your scenarios.

The curriculum is complete, but the simulator is a sandbox, not a course you finish. Open the simulator, build something, and break it.