Monedula

Kafka troubleshooting

OffsetOutOfRangeException (OFFSET_OUT_OF_RANGE)

Kafka error · OFFSET_OUT_OF_RANGE · not retriable

org.apache.kafka.clients.consumer.OffsetOutOfRangeException

The consumer asked for an offset the partition no longer has, almost always because retention deleted it while the consumer was behind or offline. With a reset policy the consumer jumps to the earliest or latest offset and the deleted records are skipped; only with auto.offset.reset=none does poll() throw OffsetOutOfRangeException, "Fetch position FetchPosition{offset=0, ...} is out of range for partition orders-0".

Reproduce it in the playground
Scenario 05.1.1 · Time-based retention (retention.ms) Seen in 2 more scenarios

The moment it happensOFFSET_OUT_OF_RANGE · reset to earliest. t+2.300s

On its next poll the group fetched from offset 0, but the log now starts at 2. auto.offset.reset=earliest seeks it to logStartOffset (2), it reads the surviving 2-3 and advances to HW=4. A lagging consumer recovered by jumping past the deleted prefix.

Open at this step: step 4, OFFSET_OUT_OF_RANGE · reset to earliest

What it means

A partition holds offsets from its log start offset to its log end. Retention (retention.ms, retention.bytes) deletes whole old segments and moves the log start forward. A fetch from an offset outside that range gets OFFSET_OUT_OF_RANGE (error code 1), and the consumer applies auto.offset.reset: earliest seeks to the log start, latest (the default) to the log end, by_duration (for example by_duration:P1D) to the offset that far back in time. It logs "Fetch position FetchPosition{offset=0, ...} is out of range for partition orders-0, resetting offset", and once it has looked up the new position, "Resetting offset for partition orders-0 to position FetchPosition{offset=1200, ...}." With none it logs "raising error to the application since no reset policy is configured" and poll() throws org.apache.kafka.clients.consumer.OffsetOutOfRangeException; the protocol error itself maps to org.apache.kafka.common.errors.OffsetOutOfRangeException. When retention caused it, the deleted records are gone for good: the group never processed them. A group with no committed offset at all, for example after offsets.retention.minutes removed it, is a different case: with none it fails with NoOffsetForPartitionException instead.

Likely causes, most common first

  1. The consumer fell behind retention. Lag grew past retention.ms (7 days by default) or retention.bytes: a slow consumer, a backlog after an outage, or retention lowered to save disk. Retention removes a segment file at a time, so the cut-off moves in steps rather than record by record.
  2. The group was offline longer than retention. The committed offset stays where the group stopped. When the group returns after the segments holding that offset were deleted, its first fetch is out of range. Scenarios 05.1.1 (reset to earliest) and 05.1.3 (auto.offset.reset=none, group stuck) show both outcomes.
  3. Total retention passed on a tiered topic. With tiered storage, a segment is copied to the remote tier once it rolls, and local.retention.ms only deletes the local copy, so a lagging consumer can still read it from the remote tier. retention.ms and retention.bytes delete from the remote tier too, and only that deletion makes an old offset out of range, as scenario 05.3.3 shows.
  4. The application seeks to an offset that does not exist. consumer.seek() to a hard-coded value, to an offset stored outside Kafka, or to one taken from another topic or cluster asks for a position the partition never had or no longer has.

How to fix it

What not to do

Does the client retry it?

Not as such. With auto.offset.reset set to earliest, latest or by_duration, the consumer resets its position and fetches again without raising anything. With auto.offset.reset=none, poll() keeps throwing until the application seeks to a valid offset or an operator resets the group's offsets.

Configs involved

Reproduce it in the simulator

Sources

Related

All troubleshooting pages