Scenario 05.1.3 · Offset out of range

Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none

Storage · Intermediate · about 2 min · Apache Kafka 4.3

Retention deletes the prefix while a group is offline; auto.offset.reset=none leaves it stuck.

What you start with

While a consumer group is offline, retention can delete the very prefix it was about to read. On rejoining, its committed offset now points before logStartOffset - an OFFSET_OUT_OF_RANGE condition. With auto.offset.reset=none the consumer has no recovery policy, so it stalls instead of jumping.

Step by step

1. Group offline · auto.offset.reset=none t+0.000s The consumer in group c1 has crashed (its member is offline) and its recovery policy is set to auto.offset.reset=none. Its committed position is offset 0. Press play. Open at this step: step 1, Group offline · auto.offset.reset=none
2. Retention deleted the prefix t+1.000s Three records were produced (HW=3), then retention advanced the partition's logStartOffset to 3 - offsets 0-2 are gone. The group is still offline at position 0, now below the log start. Open at this step: step 2, Retention deleted the prefix
3. OFFSET_OUT_OF_RANGE · group stuck t+2.200s The group restarted and polled from offset 0, but the log now starts at 3. The broker answers OFFSET_OUT_OF_RANGE, and with auto.offset.reset=none the consumer has no fallback, so poll() throws OffsetOutOfRangeException and the group is stuck. Set auto.offset.reset to earliest or latest to recover. Open at this step: step 3, OFFSET_OUT_OF_RANGE · group stuck

The takeaway

auto.offset.reset decides what happens when a committed offset no longer exists - none turns an out-of-range position into a hard stop rather than resetting to earliest or latest.

Configs in this scenario

• auto.offset.reset Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead. How auto.offset.reset works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.1.3" title="Kafka Simulator: Offset out of range" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/offset-out-of-range/">Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• OffsetOutOfRangeException in Kafka: causes and fixes

More in Storage

• Kafka retention.ms and a consumer that falls behind
• Kafka retention.bytes: size-based log retention
• Kafka log compaction: cleanup.policy=compact explained
• Kafka tombstones: deleting a key with delete.retention.ms
• Kafka cleanup.policy=compact,delete: both policies at once
• Kafka tiered storage: offloading old segments to remote
• Kafka tiered storage: a consumer reads from the remote tier
• Kafka local.retention.ms vs retention.ms in tiered storage
• Diskless Kafka topics (KIP-1163): the write path
• Diskless vs classic Kafka: cross-AZ replication traffic
• Diskless Kafka produce latency: the price of object storage

All scenarios