Monedula

Kafka troubleshooting

The consumer group keeps rebalancing

Kafka symptom

A consumer group rebalances whenever a member joins, leaves or is declared dead, so repeated rebalances mean members keep dropping out and coming back. The usual causes are processing that takes longer than max.poll.interval.ms, heartbeats missing for longer than the session timeout, and restarts of members without a static group.instance.id. A member removed for a slow poll loop logs "consumer poll timeout has expired", and its next commitSync() fails with CommitFailedException. Find which one applies before raising any timeout.

Reproduce it in the playground
Scenario 02.3.2 · Stuck consumer & max.poll.interval.ms Seen in 3 more scenarios

The moment it happensEvicted · survivor took over. t+4.000s

At t=3000 max.poll.interval.ms expired: c1's heartbeat thread left the group (max.poll.interval.ms exceeded) and p0 rebalanced onto the survivor c2, which now owns both partitions and has drained p0 back to lag 0 - the same revoke→assign rebalance a crash eviction uses.

Open at this step: step 4, Evicted · survivor took over

What it means

Each partition in a group is owned by exactly one member at a time, and the group coordinator reassigns partitions when membership changes or when a subscribed topic gains partitions. A member is removed in two ways: it stops heartbeating for longer than the session timeout, or it does not call poll() within max.poll.interval.ms, in which case the client leaves the group on its own and logs "consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages." (the KIP-848 consumer writes "Consumer poll timeout has expired."). On the classic protocol, a commitSync() from a member that was removed fails with CommitFailedException: "Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member." The KIP-848 consumer reports the same situation as "OffsetCommit failed with stale member epoch." or "OffsetCommit failed with unknown member ID." On the classic protocol, still the client default as of Kafka 4.3, every removal and every rejoin is a rebalance, and with the default RangeAssignor every member revokes all of its partitions first, so the whole group stops consuming while it runs. The consumer group protocol from KIP-848 (group.protocol=consumer, generally available since Kafka 4.0) is incremental and moves only the partitions that change owner, but a member that keeps timing out still causes churn.

Likely causes, most common first

  1. Processing takes longer than max.poll.interval.ms. If more than max.poll.interval.ms (5 minutes by default) passes between two poll() calls, the consumer leaves the group and its partitions move. Its next poll() rejoins, the same slow batch arrives again, and the cycle repeats. A commitSync() after the eviction fails with CommitFailedException, so that batch is processed a second time. A large max.poll.records (500 by default) combined with slow per-record work is the usual trigger. A static member (group.instance.id set) is the exception: it stops heartbeating but keeps its partitions until the session timeout expires, and only then are they reassigned.
  2. Heartbeats stop for longer than the session timeout. Long garbage collection pauses, CPU throttling in a container, or network trouble between the client and the group coordinator delay heartbeats. On the classic protocol the limit is session.timeout.ms (45 seconds by default, with heartbeat.interval.ms at 3 seconds). With group.protocol=consumer both are set on the broker, by group.consumer.session.timeout.ms and group.consumer.heartbeat.interval.ms.
  3. Deploys and restarts of dynamic members. Without group.instance.id, a consumer that shuts down leaves the group, and when it starts again it joins as a new member, so a rolling deploy triggers at least one rebalance per instance (KIP-345). Autoscalers that keep adding and removing consumers have the same effect.
  4. Every rebalance stops the whole group. With the eager RangeAssignor a rebalance revokes every partition from every member. That does not cause more rebalances on its own, but it turns each one into a full stop of consumption, which is usually what makes the problem visible as lag.

How to fix it

What not to do

Configs involved

Reproduce it in the simulator

Sources

Related

All troubleshooting pages