Monedula
Kafka Simulator docs menu

Run a Failure Lab experiment

Inject a fault from the Failure tab, watch the cluster react in the Inspector, Metrics and Why tabs, undo it, and repeat the experiment from a clean start.

How-to guides 6 of 10

3 min read

Updated 17 October 2026

Markdown

On this page

The Failure tab, the Failure Lab, holds every fault the simulator can inject. The steps use the free-play sandbox the playground opens with, whose topic orders has replication factor 3 and min.insync.replicas 2; on a scripted scenario, the first fault forks it into a sandbox.

Steps

1. Open the Failure tab

Select the Failure tab in the right panel. It opens on experiments: named failures such as Lose the leader, Split-brain or Drop below min.insync.replicas, each with what to watch for and the broker it will hit. Run starts one on that broker for you and Recover undoes it. This how-to uses the switches under them instead: select Show every switch. The switches are ordered by blast radius. On this single-DC cluster they start at Brokers, one row per broker with its state, healthy, slow or dead, and its controls: kill · graceful, kill · hard, slow and remove. Below come Network partition, KRaft voters, Topic config, Consumer stall (max.poll.interval.ms), Producer fault (idempotence) and the action log. On a multi-DC or two-region cluster the switches are already open, and the data-center and failover sections come first.

2. Make a follower slow

On the broker-3 row, select slow. Its tag changes to slow and the button to un-slow. A slow follower falls behind and drops out of the in-sync replicas of the partitions it follows. Select partition 0 of orders on the canvas: within a few frames its in-sync replicas are broker-1 and broker-2. The Metrics tab’s Under-replicated partitions card counts the partitions that lost broker-3. Writes still succeed.

3. Push past the limit

On the broker-2 row, select kill · hard. Partitions that kept only broker-1 and broker-2 in sync are now down to one in-sync replica, below the minimum, so the producer’s acks=all writes to them are refused and retried. The Why tab lists the partitions below min.insync.replicas among its problems, each with its in-sync count next to the minimum.

4. Undo the faults

On the broker-2 row, select restart; on the broker-3 row, un-slow. Once broker-2 is back in sync, no partition is below its minimum, writes succeed again and the Why tab stops reporting the partitions.

5. Read the action log, then start over

The action log at the foot of the tab lists what the session did, with times: the faults you injected, the records the auto-loop produced and any config change. The same list travels in a shared link. To run the experiment again from the same cluster, select Clear state above the canvas.

Other faults to try

  • Experiments: each card at the top of the tab runs a failure on the right target. Lose the leader hard-kills the broker that leads partition 0, so you do not have to look it up, and Recover restarts it.
  • kill · graceful hands leadership to an in-sync follower before the broker stops; kill · hard crashes it.
  • Network partition: tick side B on some brokers, then partition to split the cluster, and heal to reconnect it.
  • Topic config: unclean.leader.election.enable lets an out-of-sync replica take over when no in-sync one is left, and loses its missing records.
  • Consumer stall: stall a member so it stops polling until max.poll.interval.ms evicts it; resume before then to cancel the eviction.
  • Producer fault: drop ack makes the producer resend a record; an idempotent producer deduplicates it and a non-idempotent one writes a duplicate.

Check it worked

After step 4, the broker-2 and broker-3 rows read healthy again, and the newest lines of the action log include restart_broker brokerId=broker-2 and set_replica_speed brokerId=broker-3 slow=false, between the auto-loop’s produce_record lines.