---
title: "Run a Failure Lab experiment"
description: "Inject a fault from the Failure tab, watch the cluster react in the Inspector, Metrics and Why tabs, undo it, and repeat the experiment from a clean start."
url: https://monedula.dev/flock/docs/kafka-simulator/how-to/run-a-failure-lab-experiment/
---
The **Failure** tab, the Failure Lab, holds every fault the simulator can inject. The steps use the free-play sandbox the [playground](/kafka-simulator/playground/) opens with, whose topic `orders` has replication factor 3 and `min.insync.replicas` 2; on a scripted scenario, the first fault forks it into a sandbox.

## Steps

### 1. Open the Failure tab

Select the **Failure** tab in the right panel. It opens on experiments: named failures such as **Lose the leader**, **Split-brain** or **Drop below min.insync.replicas**, each with what to watch for and the broker it will hit. **Run** starts one on that broker for you and **Recover** undoes it. This how-to uses the switches under them instead: select **Show every switch**. The switches are ordered by blast radius. On this single-DC cluster they start at **Brokers**, one row per broker with its state, **healthy**, **slow** or **dead**, and its controls: **kill · graceful**, **kill · hard**, **slow** and **remove**. Below come **Network partition**, **KRaft voters**, **Topic config**, **Consumer stall (max.poll.interval.ms)**, **Producer fault (idempotence)** and the **action log**. On a multi-DC or two-region cluster the switches are already open, and the data-center and failover sections come first.

### 2. Make a follower slow

On the `broker-3` row, select **slow**. Its tag changes to **slow** and the button to **un-slow**. A slow follower falls behind and drops out of the in-sync replicas of the partitions it follows. Select partition 0 of `orders` on the canvas: within a few frames its in-sync replicas are `broker-1` and `broker-2`. The **Metrics** tab's **Under-replicated partitions** card counts the partitions that lost `broker-3`. Writes still succeed.

### 3. Push past the limit

On the `broker-2` row, select **kill · hard**. Partitions that kept only `broker-1` and `broker-2` in sync are now down to one in-sync replica, below the minimum, so the producer's `acks=all` writes to them are refused and retried. The **Why** tab lists the partitions below `min.insync.replicas` among its problems, each with its in-sync count next to the minimum.

### 4. Undo the faults

On the `broker-2` row, select **restart**; on the `broker-3` row, **un-slow**. Once `broker-2` is back in sync, no partition is below its minimum, writes succeed again and the **Why** tab stops reporting the partitions.

### 5. Read the action log, then start over

The **action log** at the foot of the tab lists what the session did, with times: the faults you injected, the records the auto-loop produced and any config change. The same list travels in a [shared link](/flock/docs/kafka-simulator/how-to/share-by-url/). To run the experiment again from the same cluster, select **Clear state** above the canvas.

## Other faults to try

- **Experiments**: each card at the top of the tab runs a failure on the right target. **Lose the leader** hard-kills the broker that leads partition 0, so you do not have to look it up, and **Recover** restarts it.
- **kill · graceful** hands leadership to an in-sync follower before the broker stops; **kill · hard** crashes it.
- **Network partition**: tick **side B** on some brokers, then **partition** to split the cluster, and **heal** to reconnect it.
- **Topic config**: **unclean.leader.election.enable** lets an out-of-sync replica take over when no in-sync one is left, and loses its missing records.
- **Consumer stall**: **stall** a member so it stops polling until `max.poll.interval.ms` evicts it; **resume** before then to cancel the eviction.
- **Producer fault**: **drop ack** makes the producer resend a record; an idempotent producer deduplicates it and a non-idempotent one writes a duplicate.

## Check it worked

After step 4, the `broker-2` and `broker-3` rows read **healthy** again, and the newest lines of the action log include `restart_broker brokerId=broker-2` and `set_replica_speed brokerId=broker-3 slow=false`, between the auto-loop's `produce_record` lines.

## Related

- [Build a cluster in free play and break it](/flock/docs/kafka-simulator/tutorials/build-a-cluster-and-break-it/) walks through broker kills from the Inspector.
- The [scenario catalog](/flock/docs/kafka-simulator/reference/scenarios/) lists the scripted chaos scenarios. They also have their own pages, such as [Single broker hard kill](/kafka-simulator/scenarios/single-broker-hard-kill/), [Network partition: split-brain](/kafka-simulator/scenarios/network-partition-split-brain/) and [Cascading failure: losing the ISR one by one](/kafka-simulator/scenarios/cascading-isr-loss/).
- The [troubleshooting pages](/kafka-simulator/troubleshooting/) start from what a fault looks like from outside, such as [Partitions are offline with no leader](/kafka-simulator/troubleshooting/offline-partitions-no-leader/), [Kafka partitions are under-replicated](/kafka-simulator/troubleshooting/under-replicated-partitions/) or [The consumer group keeps rebalancing](/kafka-simulator/troubleshooting/consumer-group-keeps-rebalancing/), and link to the scenario steps that reproduce it.