Kafka troubleshooting

NotEnoughReplicasException (NOT_ENOUGH_REPLICAS)

Kafka error · NOT_ENOUGH_REPLICAS · retriable

org.apache.kafka.common.errors.NotEnoughReplicasException

The partition has fewer in-sync replicas than its min.insync.replicas, so the leader refuses acks=all writes instead of acknowledging data that too few brokers hold. Bring the missing replicas back into the ISR; lower min.insync.replicas only as a deliberate durability trade-off.

Reproduce it in the playground

Scenario 01.1.3 · acks trade-off Seen in 6 more scenarios

The moment it happensacks=all produce blocked - NotEnoughReplicas. t+9.200s

Durability wins: acks=all refuses to ack below min.insync.replicas.

Open at this step: step 4, acks=all produce blocked - NotEnoughReplicas

What it means

With acks=all a write succeeds only once every replica in the in-sync replica set (ISR) has it, and the topic setting min.insync.replicas is the smallest ISR the leader accepts such a write with. The leader checks the ISR size before it appends: below the minimum it rejects the batch with NOT_ENOUGH_REPLICAS (error code 19) and writes nothing. The broker message gives the numbers, for example "The size of the current ISR : 1 is insufficient to satisfy the min.isr requirement of 2 for partition orders-0, ...". NotEnoughReplicasException extends RetriableException, so the producer logs a warning per attempt ("Got error produce response ... retrying ... Error: NOT_ENOUGH_REPLICAS") and retries with backoff until delivery.timeout.ms (default 2 minutes) runs out. Only then does the send fail, with this exception or with a TimeoutException ("Expiring N record(s)") if the batch expired while waiting for its next attempt. A relative, NOT_ENOUGH_REPLICAS_AFTER_APPEND (code 20), means the ISR shrank after the leader had already appended the batch, so a retry from a non-idempotent producer can write a duplicate. Writes with acks=0 or acks=1 are not checked against min.insync.replicas.

Likely causes, most common first

A broker is down or a follower fell behind. A follower that has not caught up to the leader's log end within replica.lag.time.max.ms (default 30 seconds) is removed from the ISR, and a stopped broker leaves every ISR it was in. With replication factor 3 and min.insync.replicas 2, one missing replica is tolerated; a second one blocks acks=all writes on that partition.
Several replicas failed together. Brokers that share a rack, power feed or availability zone fail as a unit. If too many of a partition's replicas share one failure domain, a single outage takes the ISR below the minimum: with replication factor 4 split over two racks and min.insync.replicas 3, losing either rack is enough.
min.insync.replicas leaves no headroom. With replication factor 3 and min.insync.replicas 3, any single replica restart, including a routine rolling restart, blocks acks=all writes. The same happens when someone raises the minimum on a live topic with kafka-configs.sh; topic configs change without a broker restart.
The topic has fewer replicas than the minimum. A topic whose replication factor is below min.insync.replicas can never take an acks=all write, and one whose replication factor equals it fails on the first lost replica. Check topics auto-created with a low default.replication.factor against a broker-wide minimum of 2.

How to fix it

List the affected partitions with kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitions, compare each partition's Isr with its Replicas, and restart or repair the brokers that are missing.
Watch the broker metrics UnderMinIsrPartitionCount, AtMinIsrPartitionCount and IsrShrinksPerSec (kafka.server:type=ReplicaManager); a partition at its minimum is one failure away from this error.
If followers are alive but lagging, look at their disk, network and replication throughput before changing replica.lag.time.max.ms.
Spread replicas over failure domains: set broker.rack on every broker so replicas of a partition land on different racks, and size min.insync.replicas so that losing one rack or zone still leaves enough in-sync replicas.
If the topic really must accept writes with fewer copies, lower it per topic, for example from 3 to 2 with kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders --alter --add-config min.insync.replicas=2, and record the durability you are giving up.

What not to do

Switching producers to acks=1 or lowering min.insync.replicas cluster-wide to make the error disappear: both trade durability for availability without fixing the missing replicas.
Raising delivery.timeout.ms or retries instead of fixing the ISR: the producer only waits longer before it fails.

Does the client retry it?

Yes. The Kafka client treats this error as retriable and retries the request until delivery.timeout.ms runs out.

Configs involved

min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS.
acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
replica.lag.time.max.ms How long a follower may go without fetching, or without catching up to the end of the leader log, before the leader removes it from the ISR (default 30000, 30 seconds).
default.replication.factor The replication factor of topics created automatically or without one (default 1); production clusters usually set 3.
broker.rack The rack or availability zone of the broker; when a topic is created, the replicas of each partition are spread across racks so one rack failing cannot take every copy.
retries How many times the producer sends a batch again after a transient error (default 2147483647, so in practice delivery.timeout.ms ends the retrying); idempotence needs it above 0.
delivery.timeout.ms The upper bound on reporting success or failure after send() returns, covering the wait in a batch, the wait for the acknowledgement and every retry (default 120000, 2 minutes); it should be at least request.timeout.ms plus linger.ms.

Reproduce it in the simulator

Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR (acks trade-off) Open at the moment it happens: Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR
Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS (Idempotence retry) Open at the moment it happens: Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS
min.insync.replicas with acks=all: when Kafka refuses writes (min.ISR boundary) Open at the moment it happens: min.insync.replicas with acks=all: when Kafka refuses writes
Change Kafka min.insync.replicas at runtime, no restart (Live config changes) Open at the moment it happens: Change Kafka min.insync.replicas at runtime, no restart
Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one (Cascading failure: losing the ISR one by one) Open at the moment it happens: Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one
Kafka rack failure: why RF=4 on two racks refuses writes (Correlated failure: a whole rack goes down) Open at the moment it happens: Kafka rack failure: why RF=4 on two racks refuses writes
Confluent MRC observer promotion: losing the Kafka leader DC (Observer promotion needs a reachable leader) Open at the moment it happens: Confluent MRC observer promotion: losing the Kafka leader DC

Sources

All troubleshooting pages