Scenario 02.1.4 · Replay: rewind to an earlier offset

Replay a Kafka topic: consumed records stay in the log

Consumer groups · Intermediate · about 3 min · Apache Kafka 4.3

The log keeps consumed records; the committed offset is just a bookmark. Reset it to earliest and the same group replays what it already read.

What you start with

Kafka does not delete records when they are consumed; the committed offset is only a bookmark into the retained log. This scenario resets a group's committed offset back to earliest (the offset-reset workflow) and watches the same group re-read records it already processed.

Step by step

1. g1 has read O1-O3 · committed offset 3 t+2.200s g1 polled all three records and auto-commit saved its position, so the committed offset is 3 - the bookmark says "resume after O3". The records are still in the log; reading them deleted nothing. Open at this step: step 1, g1 has read O1-O3 · committed offset 3
2. Group stopped to reset its offset t+2.500s To rewind a real group you first stop it: kafka-consumer-groups --reset-offsets refuses to run while members are active. Stopping alone keeps the committed offset of 3; here the group is deleted, which also deletes its offsets, so we can choose a new starting point. Open at this step: step 2, Group stopped to reset its offset
3. g1 rewound to the start · about to replay t+2.800s The same group id is back with auto.offset.reset=earliest and no committed offset, so it seeks to the log start (offset 0). Its position dropped from 3 back to 0 - it is about to re-read records it already processed. This is replay: reprocess after a bug fix, rebuild a derived view, etc. Open at this step: step 3, g1 rewound to the start · about to replay
4. Re-reading O1-O3 again t+3.200s One record per poll (max.poll.records=1), g1 works back through the same offsets it consumed before - position is 2, so O1 and O2 have been delivered a second time. The log was never consumed-away; the offset is just a movable bookmark. Open at this step: step 4, Re-reading O1-O3 again
5. Replay complete · back at offset 3 t+3.700s Having re-read O1-O3, g1's position and committed offset are 3 again - exactly where they were before the rewind, but every record was delivered twice. (Replay is at-least-once: a downstream that can't tolerate duplicates must dedupe.) Open at this step: step 5, Replay complete · back at offset 3
6. Caught up · reads the new O4 live t+5.000s After draining the replayed backlog the group is level with the log again, so the freshly produced O4 is read normally and the committed offset advances to 4. The rewind was a one-time replay, not a permanent mode. Open at this step: step 6, Caught up · reads the new O4 live

The takeaway

Because the data stays in the log, resetting the committed offset lets the same group replay history - the basis for reprocessing after a bug fix or rebuilding a derived view.

Configs in this scenario

• auto.offset.reset Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead. How auto.offset.reset works
• max.poll.records The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=02.1.4" title="Kafka Simulator: Replay: rewind to an earlier offset" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/replay-rewind-offset/">Replay a Kafka topic: consumed records stay in the log</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• Why does a Kafka consumer reprocess messages after restart?

More in Consumer groups

• Kafka consumer poll loop: subscribe, poll, fetch
• Kafka auto.offset.reset: earliest vs latest vs none
• Kafka committed offsets: enable.auto.commit and restart
• Kafka consumer group: how partitions split across members
• Kafka consumer group rebalance when a consumer joins
• Kafka partition assignors: range vs round-robin vs sticky
• Kafka eager vs cooperative rebalancing (cooperative-sticky)
• Kafka consumer crash and session.timeout.ms
• Kafka max.poll.interval.ms: evicting a stuck consumer
• Kafka static membership: group.instance.id avoids rebalances
• Kafka consumer lag: why it grows and how it drains
• Evicted Kafka consumer: how a survivor takes its partition

All scenarios