Kafka producer config
max.in.flight.requests.per.connection and message ordering
max.in.flight.requests.per.connection is how many produce requests a producer may have sent on one connection without an answer yet (default 5). Without idempotence, a value above 1 lets a retried batch land after a later one, reordering the partition. With idempotence on, the broker keeps order for up to 5 in-flight requests, so the default is both fast and ordered.
- Config
max.(producer)in. flight. requests. per. connection - Default
- 5
- Valid values
- 1 or more; 5 or less with idempotence on
What it does
The producer pipelines requests: it can send the next batch to a broker before the previous one is acknowledged. When a request fails with a retriable error and is sent again, the requests behind it may already have been written. An idempotent producer numbers every batch per partition, and the broker rejects a batch that arrives out of sequence so the producer can resend in order; the broker remembers the last 5 batches of each producer, which is where the limit of 5 comes from. A producer without idempotence has no such check, so a retry can append after a later batch.
The trade-off
More requests in flight means more throughput on a high-latency link, because the producer does not wait a full round trip per batch. Setting it to 1 is the old way to keep order without idempotence, at the cost of that pipelining. Above 5, idempotence is no longer possible: if enable.idempotence was not set explicitly it is turned off without a word, and if it was set to true the producer fails with a ConfigException.
Advice
- Keep the default of 5 with idempotence on (the default since Kafka 3.0): order and no duplicates, with pipelining.
- Only drop it to 1 if idempotence must stay off, and accept that retries can still write duplicates.
- Do not raise it above 5 for throughput; raise linger.ms or batch.size instead.
See it in the simulator
- Kafka max.in.flight.requests.per.connection and ordering (Out-of-order detection and max.in.flight) Step 5: A's retry lands at offset 2 - REORDER Open at this step: Kafka max.in.flight.requests.per.connection and ordering