Scenario 02.3.4 · Sustained consumer lag
Kafka consumer lag: why it grows and how it drains
A consumer capped by max.poll.records cannot keep up with a faster producer, so lag accumulates - then drains once the producer stops.
Run this scenario in the playgroundReady · lag 0. t+0.000s
A single consumer with max.poll.records = 2 polls every 600 ms. The producer is about to write ~5 records/sec - faster than the consumer can drain. Press play.
Open at this step: step 1, Ready · lag 0Falling behind. t+0.700s
Each poll returns at most 2 records, but the producer keeps appending. The committed offset trails the high watermark - lag is building.
Open at this step: step 2, Falling behindSustained lag. t+1.900s
The 2-record poll cap can't match the producer's rate, so lag holds near its peak while the producer stays ahead. This is the core lesson: a consumer slower than its producer accumulates sustained lag.
Open at this step: step 3, Sustained lagDrained · caught up. t+4.300s
The producer has stopped. With no new records arriving, the capped consumer finally drains the backlog two at a time and lag returns to 0.
Open at this step: step 4, Drained · caught upWhat you start with
Consumer lag is the gap between the end of the log and a group's position. Here max.poll.records caps the consumer's throughput below the produce rate, so lag steadily accumulates while the producer runs, then drains once it stops.
Step by step
- Ready · lag 0 t+0.000s A single consumer with max.poll.records = 2 polls every 600 ms. The producer is about to write ~5 records/sec - faster than the consumer can drain. Press play. Open at this step: step 1, Ready · lag 0
- Falling behind t+0.700s Each poll returns at most 2 records, but the producer keeps appending. The committed offset trails the high watermark - lag is building. Open at this step: step 2, Falling behind
- Sustained lag t+1.900s The 2-record poll cap can't match the producer's rate, so lag holds near its peak while the producer stays ahead. This is the core lesson: a consumer slower than its producer accumulates sustained lag. Open at this step: step 3, Sustained lag
- Drained · caught up t+4.300s The producer has stopped. With no new records arriving, the capped consumer finally drains the backlog two at a time and lag returns to 0. Open at this step: step 4, Drained · caught up
The takeaway
Lag grows whenever consume rate trails produce rate and only shrinks when the consumer catches up - the core signal for whether you need to scale consumers or speed up processing.
Configs in this scenario
-
max.The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.poll. records
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=02.3.4" title="Kafka Simulator: Sustained consumer lag" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/sustained-lag/">Kafka consumer lag: why it grows and how it drains</a></p>Further reading
Troubleshooting
More in Consumer groups
- Kafka consumer poll loop: subscribe, poll, fetch
- Kafka auto.offset.reset: earliest vs latest vs none
- Kafka committed offsets: enable.auto.commit and restart
- Replay a Kafka topic: consumed records stay in the log
- Kafka consumer group: how partitions split across members
- Kafka consumer group rebalance when a consumer joins
- Kafka partition assignors: range vs round-robin vs sticky
- Kafka eager vs cooperative rebalancing (cooperative-sticky)
- Kafka consumer crash and session.timeout.ms
- Kafka max.poll.interval.ms: evicting a stuck consumer
- Kafka static membership: group.instance.id avoids rebalances
- Evicted Kafka consumer: how a survivor takes its partition