Kafka consumer config

max.poll.interval.ms: when slow processing triggers a rebalance

Consumer config · default 300000 (5 minutes) · Apache Kafka 4.3

max.poll.interval.ms is the longest a consumer may go between two poll() calls before it counts as failed (default 300000, 5 minutes). When processing a batch takes longer, the consumer leaves the group, its partitions move to other members, and its next offset commit fails with CommitFailedException. It catches a consumer that is alive but stuck, which heartbeats alone cannot.

Config: max.poll.interval.ms (consumer)
Default: 300000 (5 minutes)
Valid values: 1 or more (milliseconds)
Reference: Apache Kafka documentation for max.poll.interval.ms

Scenario 02.3.2 · Stuck consumer & max.poll.interval.ms step 3

c1 stuck processing · p0 stalls, c2 keeps consuming. c1 got stuck processing at t=1000. It still heartbeats (its session never times out) and still owns p0, but it no longer polls - so new records on p0 are NOT consumed and its lag grows, while c2 keeps draining p1. Unlike a crash, the member is alive: this is the max.poll.interval.ms failure, not session.timeout.ms. Open this step in the full playground

What it does

Heartbeats come from a background thread, so a consumer whose processing loop hangs would otherwise keep its partitions forever. The consumer checks how long ago the application last called poll(), and when that passes max.poll.interval.ms, its heartbeat thread sends a request to leave the group. The group rebalances and another member continues from the last committed offset, so the records the slow consumer processed without committing are processed again. A static member (group.instance.id) does not leave: its partitions stay with it until session.timeout.ms runs out.

The trade-off

A longer interval tolerates slow batches, but a consumer that is truly stuck holds its partitions that much longer. A shorter one reassigns stuck partitions sooner, but a slow batch in normal operation causes a rebalance loop: the member leaves, rejoins on its next poll, and the group rebalances again and again.

Advice

First lower max.poll.records (default 500) so each poll() returns less work; that is the usual fix.
Raise max.poll.interval.ms only to the worst-case processing time of one batch, with margin.
Move slow work, such as calls to other services, off the poll loop, or pause the partitions while it runs and keep calling poll().

See it in the simulator

Kafka max.poll.interval.ms: evicting a stuck consumer (Stuck consumer & max.poll.interval.ms) Step 3: c1 stuck processing · p0 stalls, c2 keeps consuming Open at this step: Kafka max.poll.interval.ms: evicting a stuck consumer

Related configs

All Kafka configs