---
title: "Kafka messages lost after broker failure: causes and fixes"
description: "Why acknowledged Kafka records vanish when a broker dies: acks=1, unclean leader election, an ISR of one, ignored send errors, and the settings that prevent it."
url: https://monedula.dev/kafka-simulator/troubleshooting/messages-lost-after-broker-failure/
---

# Messages are lost after a broker failure

Kafka symptom.

## Quick answer

Kafka only promises that a committed record, one that every in-sync replica has, survives as long as one in-sync replica stays alive. Records are lost after a broker failure when they were acknowledged before that point (acks=0 or acks=1), when a replica outside the ISR became leader (unclean leader election), or when the ISR had already shrunk to a single broker. Produce with acks=all to topics with replication factor 3 and min.insync.replicas=2, and keep unclean leader election off.

## What it means

A record becomes committed once every replica in the ISR has it, and only committed records are given to consumers. When a leader fails, the controller picks the new leader from the ISR, so the new leader has every committed record. When the old leader comes back as a follower, it truncates whatever the new leader does not have before it fetches again, and logs "Truncating to offset 120" for the partition. Whether your producer was told about such a record decides whether it was lost or simply retried: acks=all is acknowledged only once the record is committed, acks=1 as soon as the leader has written it, and acks=0 never. Since Kafka 3.0 producers default to acks=all with idempotence on, but the broker defaults still allow loss: min.insync.replicas=1 lets acks=all succeed on the leader alone, and default.replication.factor=1 creates single-copy topics. Most loss comes from those defaults left in place, from producer settings or topic overrides someone changed, or from applications that ignore failed sends.

## Likely causes, most common first

1. **The producer used acks=1 or acks=0.** With acks=1 the leader acknowledges after writing to its own log. If it dies before a follower fetches the record, the new leader never had it. acks=0 does not wait at all, and the producer is never told. min.insync.replicas does not protect either one: it is only checked for acks=all.
2. **Unclean leader election promoted a stale replica.** With unclean.leader.election.enable=true, on the broker or as a topic override, a replica outside the ISR can become leader when no in-sync replica is available. Its log becomes the source of truth, and records it never received are gone for good, including ones acks=all producers were told had been committed. The default is false.
3. **The ISR had shrunk to the leader alone.** acks=all waits for the current ISR, not for the full replica list. With min.insync.replicas=1, which is the broker default, a partition whose followers have fallen out keeps acknowledging writes that exist on one broker, and losing that broker loses them.
4. **All replicas failed together.** Replicas placed in one rack, zone or power domain can go down at the same time. Without broker.rack set, replica placement does not take that into account.
5. **The send failed and nobody looked.** send() is asynchronous. If a record cannot be delivered within delivery.timeout.ms (2 minutes by default), the error arrives in the callback or the returned Future, and code that checks neither drops the record without a trace.

## How to fix it

- Produce with acks=all (the default since Kafka 3.0) to topics with replication factor 3 and min.insync.replicas=2, so every acknowledged record exists on at least two brokers.
- Keep unclean.leader.election.enable=false on the brokers and check topic overrides with kafka-configs.sh --describe. Alert on the controller metric UncleanLeaderElectionsPerSec, which should stay at 0.
- Handle send failures in the producer callback: log them, alert on them, and stop or retry deliberately instead of carrying on.
- Set broker.rack on every broker so the replicas of each partition land in different racks or zones.

## What not to do

- Enabling unclean leader election to bring a partition back online without deciding that losing its recent records is acceptable. If that is the decision, run it for the one partition with kafka-leader-election.sh --election-type unclean rather than turning it on for the whole cluster.

## Configs involved

- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS.
- [`unclean.leader.election.enable`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_unclean.leader.election.enable): Lets a replica that is not in the ISR become leader when no in-sync replica is left, which brings the partition back online but loses the writes that replica never received (default false).
- [`enable.idempotence`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_enable.idempotence): Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off.
- [`delivery.timeout.ms`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_delivery.timeout.ms): The upper bound on reporting success or failure after send() returns, covering the wait in a batch, the wait for the acknowledgement and every retry (default 120000, 2 minutes); it should be at least request.timeout.ms plus linger.ms.
- [`broker.rack`](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_broker.rack): The rack or availability zone of the broker; when a topic is created, the replicas of each partition are spread across racks so one rack failing cannot take every copy.

## Reproduce it in the simulator

- [Kafka acks=1 data loss: min.insync.replicas does not help](https://monedula.dev/kafka-simulator/scenarios/acks1-durability-gap/) (acks=1 durability gap): [open at the moment it happens](https://monedula.dev/kafka-simulator/playground/?scenario=01.1.4&t=4)
- [Kafka producer acks=0: fire and forget, silent data loss](https://monedula.dev/kafka-simulator/scenarios/acks-0-fire-and-forget/) (acks=0: fire and forget): [open at the moment it happens](https://monedula.dev/kafka-simulator/playground/?scenario=01.1.1&t=3)
- [Kafka unclean leader election: permanent loss of acked data](https://monedula.dev/kafka-simulator/scenarios/unclean-election-data-loss/) (Unclean election: permanent data loss): [open at the moment it happens](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=4)

## Sources

- [Apache Kafka 4.3: producer config acks](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks)
- [Apache Kafka 4.3: topic config min.insync.replicas](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas)
- [Apache Kafka 4.3: topic config unclean.leader.election.enable](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_unclean.leader.election.enable)
- [Apache Kafka 4.3: design, replication and committed messages](https://kafka.apache.org/43/design/design/#replication)
- [Apache Kafka 4.3: design, availability and durability guarantees](https://kafka.apache.org/43/design/design/#availability-and-durability-guarantees)
- [Apache Kafka 4.3: broker config default.replication.factor (default 1)](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_default.replication.factor)
- [Apache Kafka 4.3: producer config delivery.timeout.ms](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_delivery.timeout.ms)
- [Apache Kafka 4.3: balancing replicas across racks](https://kafka.apache.org/43/operations/basic-kafka-operations/#balancing-replicas-across-racks)
- [Apache Kafka 4.3: monitoring (UncleanLeaderElectionsPerSec)](https://kafka.apache.org/43/operations/monitoring/)
- [KIP-101: truncation by leader epoch](https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation)
- [KIP-679: producer enables the strongest delivery guarantee by default](https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default)
- [Apache Kafka 4.3.0 source: UnifiedLog.java (Truncating to offset)](https://github.com/apache/kafka/blob/4.3.0/storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java)

## Related

- [Kafka offline partitions with no leader: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/offline-partitions-no-leader/)
- [Kafka under-replicated partitions: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/under-replicated-partitions/)