Monedula

Kafka troubleshooting

Messages are lost after a broker failure

Kafka symptom

Kafka only promises that a committed record, one that every in-sync replica has, survives as long as one in-sync replica stays alive. Records are lost after a broker failure when they were acknowledged before that point (acks=0 or acks=1), when a replica outside the ISR became leader (unclean leader election), or when the ISR had already shrunk to a single broker. Produce with acks=all to topics with replication factor 3 and min.insync.replicas=2, and keep unclean leader election off.

Reproduce it in the playground
Scenario 01.1.4 · acks=1 durability gap Seen in 2 more scenarios

The moment it happensNew leader at LEO 0 - record lost. t+6.000s

A new leader is elected from the surviving ISR at LEO 0. There is nothing to recover: the only copy was on the dead leader. The high watermark is still 0 and never advances.

Open at this step: step 5, New leader at LEO 0 - record lost

What it means

A record becomes committed once every replica in the ISR has it, and only committed records are given to consumers. When a leader fails, the controller picks the new leader from the ISR, so the new leader has every committed record. When the old leader comes back as a follower, it truncates whatever the new leader does not have before it fetches again, and logs "Truncating to offset 120" for the partition. Whether your producer was told about such a record decides whether it was lost or simply retried: acks=all is acknowledged only once the record is committed, acks=1 as soon as the leader has written it, and acks=0 never. Since Kafka 3.0 producers default to acks=all with idempotence on, but the broker defaults still allow loss: min.insync.replicas=1 lets acks=all succeed on the leader alone, and default.replication.factor=1 creates single-copy topics. Most loss comes from those defaults left in place, from producer settings or topic overrides someone changed, or from applications that ignore failed sends.

Likely causes, most common first

  1. The producer used acks=1 or acks=0. With acks=1 the leader acknowledges after writing to its own log. If it dies before a follower fetches the record, the new leader never had it. acks=0 does not wait at all, and the producer is never told. min.insync.replicas does not protect either one: it is only checked for acks=all.
  2. Unclean leader election promoted a stale replica. With unclean.leader.election.enable=true, on the broker or as a topic override, a replica outside the ISR can become leader when no in-sync replica is available. Its log becomes the source of truth, and records it never received are gone for good, including ones acks=all producers were told had been committed. The default is false.
  3. The ISR had shrunk to the leader alone. acks=all waits for the current ISR, not for the full replica list. With min.insync.replicas=1, which is the broker default, a partition whose followers have fallen out keeps acknowledging writes that exist on one broker, and losing that broker loses them.
  4. All replicas failed together. Replicas placed in one rack, zone or power domain can go down at the same time. Without broker.rack set, replica placement does not take that into account.
  5. The send failed and nobody looked. send() is asynchronous. If a record cannot be delivered within delivery.timeout.ms (2 minutes by default), the error arrives in the callback or the returned Future, and code that checks neither drops the record without a trace.

How to fix it

What not to do

Configs involved

Reproduce it in the simulator

Sources

Related

All troubleshooting pages