Scenario 02.1.1 · Subscribe, poll, fetch loop
Kafka consumer poll loop: subscribe, poll, fetch
A consumer subscribes and polls in a loop; each poll fetches new committed records and advances its position.
Run this scenario in the playgroundConsumer subscribed · position 0. t+0.000s
The consumer c1 has subscribed to 'orders' and is polling in a loop. The log is empty and its position is 0.
Open at this step: step 1, Consumer subscribed · position 0First poll fetches O1 · position 1. t+0.900s
O1 was produced and committed; the next poll fetched it and advanced the consumer's position to 1. This poll → fetch → advance step is the consumer's fetch loop.
Open at this step: step 2, First poll fetches O1 · position 1Keeping up with the stream. t+1.700s
As more records commit, each poll fetches them and advances the position. The consumer reads only committed records (offsets below the high watermark).
Open at this step: step 3, Keeping up with the streamCaught up · position 3. t+2.400s
All three records have been fetched (position 3). With nothing new to read, further polls return empty until the producer writes again.
Open at this step: step 4, Caught up · position 3What you start with
A Kafka consumer is not pushed a stream - it drives a poll loop. Here a single consumer subscribes to a topic and repeatedly polls; each poll fetches the next batch of committed records from the broker and advances the consumer's in-memory position.
Step by step
- Consumer subscribed · position 0 t+0.000s The consumer c1 has subscribed to 'orders' and is polling in a loop. The log is empty and its position is 0. Open at this step: step 1, Consumer subscribed · position 0
- First poll fetches O1 · position 1 t+0.900s O1 was produced and committed; the next poll fetched it and advanced the consumer's position to 1. This poll → fetch → advance step is the consumer's fetch loop. Open at this step: step 2, First poll fetches O1 · position 1
- Keeping up with the stream t+1.700s As more records commit, each poll fetches them and advances the position. The consumer reads only committed records (offsets below the high watermark). Open at this step: step 3, Keeping up with the stream
- Caught up · position 3 t+2.400s All three records have been fetched (position 3). With nothing new to read, further polls return empty until the producer writes again. Open at this step: step 4, Caught up · position 3
The takeaway
Consumption is pull-based and stateful: the position moves forward one fetched batch at a time, and a consumer only ever sees records up to the partition's high-watermark.
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=02.1.1" title="Kafka Simulator: Subscribe, poll, fetch loop" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/subscribe-poll-fetch-loop/">Kafka consumer poll loop: subscribe, poll, fetch</a></p>Further reading
More in Consumer groups
- Kafka auto.offset.reset: earliest vs latest vs none
- Kafka committed offsets: enable.auto.commit and restart
- Replay a Kafka topic: consumed records stay in the log
- Kafka consumer group: how partitions split across members
- Kafka consumer group rebalance when a consumer joins
- Kafka partition assignors: range vs round-robin vs sticky
- Kafka eager vs cooperative rebalancing (cooperative-sticky)
- Kafka consumer crash and session.timeout.ms
- Kafka max.poll.interval.ms: evicting a stuck consumer
- Kafka static membership: group.instance.id avoids rebalances
- Kafka consumer lag: why it grows and how it drains
- Evicted Kafka consumer: how a survivor takes its partition