Scenario 02.1.2 · auto.offset.reset

Kafka auto.offset.reset: earliest vs latest vs none

Consumer groups · Intermediate · about 3 min · Apache Kafka 4.3

Where a new group with no committed offset starts: earliest (replay the log), latest (only new records), or none (error); Kafka 4.x adds by_duration.

What you start with

When a brand-new consumer group has no committed offset, auto.offset.reset decides where it begins reading. This scenario contrasts three of the settings - earliest replays the whole log, latest skips to only new records, and none raises an error (4.x also offers by_duration).

Step by step

1. Three records in the log · no consumers yet t+0.900s The log already holds offsets 0-2. Two fresh consumer groups are about to join - neither has a committed offset, so auto.offset.reset decides where each starts. Open at this step: step 1, Three records in the log · no consumers yet
2. g-latest starts at the END · g-earliest at the START t+1.200s Both groups just joined. g-latest (reset=latest) seeked to the END - position 3, so it will read none of O1-O3. g-earliest (reset=earliest) seeked to the START - position 0, about to replay all three. Open at this step: step 2, g-latest starts at the END · g-earliest at the START
3. g-earliest has replayed all three t+2.300s g-earliest worked through the backlog to position 3, consuming O1, O2, O3 one per poll. g-latest is still at 3 having consumed none of them - same position number, opposite history. Open at this step: step 3, g-earliest has replayed all three
4. A new record · both groups read it t+3.000s O4 arrives at offset 3 and both groups read it (position 4). Earliest replayed the backlog; latest only ever sees records produced after it joined. (reset=none would have thrown instead of choosing.) Open at this step: step 4, A new record · both groups read it

The takeaway

auto.offset.reset only applies when there is no valid committed offset: choose earliest to reprocess history, latest to skip the backlog, or none to fail fast rather than silently guess.

Configs in this scenario

• auto.offset.reset Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead. How auto.offset.reset works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=02.1.2" title="Kafka Simulator: auto.offset.reset" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/auto-offset-reset/">Kafka auto.offset.reset: earliest vs latest vs none</a></p>

Further reading

Apache Kafka documentation

More in Consumer groups

• Kafka consumer poll loop: subscribe, poll, fetch
• Kafka committed offsets: enable.auto.commit and restart
• Replay a Kafka topic: consumed records stay in the log
• Kafka consumer group: how partitions split across members
• Kafka consumer group rebalance when a consumer joins
• Kafka partition assignors: range vs round-robin vs sticky
• Kafka eager vs cooperative rebalancing (cooperative-sticky)
• Kafka consumer crash and session.timeout.ms
• Kafka max.poll.interval.ms: evicting a stuck consumer
• Kafka static membership: group.instance.id avoids rebalances
• Kafka consumer lag: why it grows and how it drains
• Evicted Kafka consumer: how a survivor takes its partition

All scenarios