Kafka troubleshooting

Consumer lag keeps growing

Kafka symptom

Lag is the distance between a partition's log end offset and the group's committed offset, so lag that only grows means the committed offset advances more slowly than producers write. First check whether every partition grows or only a few: all of them points to throughput, a few points to a stuck member, a hot key, or commits that stopped.

Reproduce it in the playground

Scenario 02.3.4 · Sustained consumer lag Seen in 2 more scenarios

The moment it happensSustained lag. t+1.900s

The 2-record poll cap can't match the producer's rate, so lag holds near its peak while the producer stays ahead. This is the core lesson: a consumer slower than its producer accumulates sustained lag.

Open at this step: step 3, Sustained lag

What it means

kafka-consumer-groups.sh --describe prints one row per partition with CURRENT-OFFSET (the group's committed offset), LOG-END-OFFSET and LAG, the difference between the two. When no consumer of the group is running, it prints "Consumer group 'orders-app' has no active members." first, and nothing drains the lag. The consumer's own records-lag-max metric measures the same distance from its current position instead of the committed offset, so the two can disagree: if records-lag-max stays small while the LAG column climbs, the consumer is reading but its commits are not landing. Lag that rises during a burst and drains afterwards is a backlog being worked off. Lag that climbs for as long as producers write means the consume rate is below the produce rate, and it does not recover on its own.

Likely causes, most common first

Consumers are slower than producers. Per-record work is too slow for the incoming rate, typically a synchronous call to a database or an HTTP service for every record. Lag then grows on every partition at a similar rate.
The group cannot use more consumers. Each partition is read by exactly one member of a group, so a group never uses more consumers than the topic has partitions. Members beyond the partition count sit idle, and adding them does not reduce lag.
A stuck or dead member still owns partitions. A consumer blocked in processing keeps heartbeating and keeps its partitions until max.poll.interval.ms expires, when it logs "consumer poll timeout has expired" and leaves the group; a crashed one keeps them until the session timeout. Until then only that member's partitions grow, while the rest of the group stays level.
One partition gets most of the traffic. Records with the same key always go to the same partition. A skewed key distribution puts most writes on one or two partitions, and the single consumer that owns each of them cannot keep up.
Commits stopped. With enable.auto.commit=false and no commit calls, or with commits failing after evictions, the committed offset stays put while records are processed. The LAG column grows, and a restart reprocesses everything since the last successful commit.

How to fix it

Run kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group a few times and note which partitions grow. Add --members --verbose to see which member owns each partition.
If every partition grows, make processing faster (batch the writes, use asynchronous I/O), then scale consumers up to the partition count. If the group is already at the partition count, add partitions, keeping in mind that this changes which partition each key maps to.
If only a few partitions grow, look at their owner. The client metrics time-between-poll-max and last-poll-seconds-ago show a consumer that has stopped polling; fix or restart it. For a hot key, change the key or split the work it carries.
If records-lag-max is low but the LAG column grows, fix the commit path: commit after processing, and log commit failures instead of ignoring them.
Alert on lag that keeps growing over a window rather than on a fixed number, so a large backlog that is shrinking does not page anyone.

What not to do

Resetting the group to the end of the log to make the number go away. kafka-consumer-groups.sh --reset-offsets --to-latest --execute skips every record in between for that group.
Raising max.poll.records when processing is the bottleneck. Bigger batches take longer between polls and can push the consumer past max.poll.interval.ms, which adds rebalances on top of the lag.

Configs involved

max.poll.records The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.
max.poll.interval.ms The longest a group member may go between poll() calls before it counts as failed and the group rebalances its partitions to other members (default 300000, 5 minutes); a static member keeps them until its session times out.
enable.auto.commit Whether the consumer commits its offsets in the background every auto.commit.interval.ms (default true); with it on, records processed between the last commit and a crash are read again.
session.timeout.ms How long the group coordinator waits for a heartbeat from a classic-protocol consumer before it removes the member and rebalances (default 45000, 45 seconds); with group.protocol=consumer the broker group.consumer.session.timeout.ms applies instead.

Reproduce it in the simulator

Kafka consumer lag: why it grows and how it drains (Sustained consumer lag) Open at the moment it happens: Kafka consumer lag: why it grows and how it drains
Diagnose consumer lag with kafka-consumer-groups --describe (Diagnose lag before opening the inspector) Open at the moment it happens: Diagnose consumer lag with kafka-consumer-groups --describe
Kafka max.poll.interval.ms: evicting a stuck consumer (Stuck consumer & max.poll.interval.ms) Open at the moment it happens: Kafka max.poll.interval.ms: evicting a stuck consumer

Sources

All troubleshooting pages