Monedula

Kafka troubleshooting

OutOfOrderSequenceException (OUT_OF_ORDER_SEQUENCE_NUMBER)

Kafka error · OUT_OF_ORDER_SEQUENCE_NUMBER · not retriable

org.apache.kafka.common.errors.OutOfOrderSequenceException

The partition leader received a batch whose sequence number does not follow the last one it stored for this producer, so it refused to append it rather than leave a gap. It usually means the broker lost records the producer had already written. Find that loss first; for a transactional producer, closing and recreating it is the safe response.

Reproduce it in the playground
Scenario 01.3.3 · Idempotent dedup vs duplicate

The moment it happensDropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops). t+11.100s

A batch the broker never received left a gap after the last accepted sequence, so the broker rejects the next batch with OUT_OF_ORDER_SEQUENCE_NUMBER. The simulator stops the producer here, as a transactional producer would have to; a Kafka 4.x idempotent producer instead bumps its epoch, resets the sequence and retries on its own.

Open at this step: step 6, Dropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops)

What it means

An idempotent producer, the default when no conflicting settings are given, stamps every batch with its producer id, an epoch and a sequence number per partition. The leader remembers the last sequences it accepted and appends a batch only if its sequence is the next one: a duplicate is acknowledged without being written again, and a gap is rejected with OUT_OF_ORDER_SEQUENCE_NUMBER (error code 45). The Javadoc spells out what that implies: data may have been lost. The leader logs the rejection at ERROR, "Error processing append operation on partition ...orders-0", with the exception "org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1000 at offset 42 in partition orders-0: 7 (incoming seq. number), 4 (current end sequence number)". An idempotent producer without a transactional.id logs "The broker returned org.apache.kafka.common.errors.OutOfOrderSequenceException: ... for topic-partition orders-0 with producerId 1000, epoch 0, and sequence number 7". The producer first retries batches that were out of order only because an earlier one was still in flight. When the gap is real, an idempotent producer without a transactional.id bumps its epoch and restarts that partition at sequence 0, so it keeps sending, but records that were pending may land out of order. For a transactional producer the KafkaProducer Javadoc says the only option left is close(); the 4.3 client treats the error as abortable, but closing is the safe advice. UnknownProducerIdException, the error for a producer id the broker has no state for, is a subclass, so a catch block for OutOfOrderSequenceException catches it too.

Likely causes, most common first

  1. The leader lost acknowledged records. After a leader change the new leader may lack batches the old one acknowledged: with acks=1, or acks=all and min.insync.replicas=1, a write can be acknowledged by the leader alone, and an unclean leader election (unclean.leader.election.enable=true) can elect a replica that is missing records. The producer's next sequence then sits beyond the new leader's last one.
  2. A batch expired while later ones were in flight. With up to five requests in flight per connection, a batch that hits delivery.timeout.ms after it was sent leaves the producer unsure whether the broker stored it. The batches already in flight behind it may then no longer line up with what the broker holds, and the producer waits for them to resolve before it sends more to that partition.

How to fix it

What not to do

Does the client retry it?

Not as the same request. The exception is not a RetriableException, but an idempotent producer in Kafka 4.x usually recovers by itself: it bumps its epoch, restarts the partition's sequence at 0 and sends again. For a transactional producer the safe response is to close it and create a new one, as the Javadoc advises, although the 4.3 client itself only marks the transaction as abortable.

Configs involved

Reproduce it in the simulator

Sources

Related

All troubleshooting pages