Kafka troubleshooting
Messages arrive out of order
Kafka guarantees order only within a partition: a consumer reads each partition in the order its records were written, and there is no order across partitions. Messages arrive out of order when related records land in different partitions, when the partition count changes, when a non-idempotent producer retries with more than one request in flight, or when the consumer processes records in parallel.
Reproduce it in the playground3 steps beforeA committed at offset 0. t+1.100s
A is produced and commits durably at offset 0 (acks=all). Its ack is now on the way back to the producer.
Open at this step: step 2, A committed at offset 02 steps beforeA's ack is lost - p1 retries A (deferred). t+3.000s
drop_ack loses A's acknowledgement, so p1 must re-send A. Because max.in.flight > 1, the connection still accepts other requests while this retry waits out retry.backoff.ms - leaving a window for a later batch to slip ahead.
Open at this step: step 3, A's ack is lost - p1 retries A (deferred)1 step beforeB commits at offset 1 - ahead of the retried A. t+3.300s
B is produced during the retry window and commits at offset 1, AHEAD of A's still-pending retry. The log is now [A, B] - but A's retry has not landed yet.
Open at this step: step 4, B commits at offset 1 - ahead of the retried AThe moment it happensA's retry lands at offset 2 - REORDER. t+3.600s
A's retry finally appends at offset 2. The log is [A, B, A]: A was produced before B, yet its retried copy sits AFTER B. max.in.flight > 1 + idempotence off reordered the log (and, being non-idempotent, also duplicated A).
Open at this step: step 5, A's retry lands at offset 2 - REORDERRight aftermax.in.flight → 1. t+5.000s
A config change drops max.in.flight.requests.per.connection to 1 (live in the simulator; a real producer is recreated with it). Now the connection holds a single in-flight request at a time, so the producer blocks later sends until each retry commits.
Open at this step: step 6, max.in.flight → 1What it means
The default partitioner sends records with the same key to the same partition, which is how per-key order survives on a topic with many partitions. Records without a key are spread across partitions batch by batch, so two of them can be read in either order. Within a partition, the producer may keep up to max.in.flight.requests.per.connection unacknowledged requests on a connection (5 by default). With the idempotent producer, on by default since Kafka 3.0, the broker uses sequence numbers to keep those batches in order for any allowed value. With idempotence off and retries on, a batch that fails and is retried can land after a batch that was sent later. The producer logs every such retry, for example "Got error produce response with correlation id 7 on topic-partition orders-0, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 1", so that warning on a producer with enable.idempotence=false and more than one request in flight is the sign to look for.
Likely causes, most common first
- Related records have different keys, or none. The order between, say, an order-created and an order-paid event only holds if both carry the same key. Keyless records, or a key that differs between event types, put them in different partitions, and partitions are read independently.
- The partition count changed. The default partitioner maps a key to a partition by its hash and the partition count, so adding partitions moves keys to different partitions. New records for a key can then be read before its older records in the old partition have been consumed.
- Retries with idempotence off. With enable.idempotence=false, retries enabled and max.in.flight.requests.per.connection above 1, a retried batch can be written after a batch sent after it. If the first attempt had in fact succeeded, the record is duplicated as well.
- Several writers, or resends from application code. Two producer instances writing the same key have no order between them; the leader appends whatever arrives first. Code that resends a failed record from its callback appends it after everything sent in the meantime.
- The consumer processes in parallel. Handing the records of one partition to a thread pool, or to asynchronous calls that complete in any order, gives up the order in which Kafka delivered them.
How to fix it
- Key every record by the entity whose order matters, such as the order id or the account id, so all of its events share a partition.
- Keep enable.idempotence=true. If it has to be off, set max.in.flight.requests.per.connection=1 so retries cannot overtake later batches, and accept the lower throughput.
- Choose the partition count of a keyed topic up front. If it has to grow, plan for a window in which a key's old and new records sit in different partitions, or move to a new topic.
- Write each key from a single producer, and let the producer retry instead of resending from application code.
- On the consumer, process each partition, or each key, in sequence. With a worker pool, route records by key so one key always goes to the same worker.
What not to do
- Moving everything to a single partition to get a total order. That limits the topic to one consumer per group and one leader broker, when per-key order is usually what the application needs.
- Setting max.in.flight.requests.per.connection=1 on an idempotent producer. Order is already preserved for any allowed value, so it only costs throughput.
Configs involved
enable.Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off.idempotence max.How many unacknowledged requests the producer sends on one connection (default 5); above 1 without idempotence a retried batch can land after a later one, and idempotence needs 5 or fewer.in. flight. requests. per. connection retriesHow many times the producer sends a batch again after a transient error (default 2147483647, so in practice delivery.timeout.ms ends the retrying); idempotence needs it above 0.partitioner.Picks the partition when the application names none: unset (the default), a record with a key goes to the partition its key hashes to and a keyless record sticks to one partition until about batch.size bytes went there, while RoundRobinPartitioner spreads records across partitions even when they have keys.class acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
Reproduce it in the simulator
- Kafka max.in.flight.requests.per.connection and ordering (Out-of-order detection and max.in.flight) Open at the moment it happens: Kafka max.in.flight.requests.per.connection and ordering
Sources
- Apache Kafka 4.3: introduction, ordering within a partition
- Apache Kafka 4.3: producer config max.in.flight.requests.per.connection
- Apache Kafka 4.3: producer config enable.idempotence
- Apache Kafka 4.3: producer config retries
- Apache Kafka 4.3: producer config partitioner.class
- Apache Kafka 4.3: modifying topics (adding partitions changes key mapping)
- Apache Kafka 4.3 Javadoc: KafkaProducer (idempotence default since 3.0)
- Apache Kafka 4.3.0 source: Sender.java (Got error produce response, retrying)