Scenario 13.1.2 · Diagnose lag before opening the inspector
Diagnose consumer lag with kafka-consumer-groups --describe
One stalled consumer, one climbing LAG column: find it with kafka-consumer-groups, name the member with --members, and keep the table on screen with watch.
Run this scenario in the playgroundA group of two, nothing produced yet. t+0.000s
Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). Nothing has been produced yet, so nobody can be behind: `kafka-consumer-groups --list` shows the one group, and `kafka-consumer-groups --describe --group g1` already prints a row per partition. Neither row has a committed offset yet, so its LAG reads "-", not 0: in this table "nothing committed" and "nothing to catch up on" are different answers.
Open at this step: step 1, A group of two, nothing produced yetSix records in, both members keeping up. t+3.200s
Both members are polling. `kafka-consumer-groups --describe --group g1` gives you one row per partition, each with a real CONSUMER-ID and a LAG at or near zero. A lone 1 there is not a problem: LAG is measured against the COMMITTED offset and auto-commit lands on the NEXT poll, so a record that has just been read still counts for one more tick.
Open at this step: step 2, Six records in, both members keeping upOne member stopped polling. t+5.200s
c2 got stuck in record processing while the producer kept appending. `kafka-consumer-groups --describe --group g1` now shows a "-" where one partition's consumer used to be, and that row's LAG pulling away from its sibling. Keep it on screen with `watch -n 1 kafka-consumer-groups --describe --group g1`: you get a dim header line carrying the command and the SIM clock, and the table under it rewritten in place as LAG climbs. The `-n 1` only decorates that header, so there is no wall clock to wait for. The re-render rides the displayed frames, so pausing the scenario freezes the header and the table with it.
Open at this step: step 3, One member stopped pollingWhich member is it? t+6.800s
The gap has widened. `kafka-consumer-groups --describe --group g1` shows it, but the lag table names the PARTITION, not the member, so `kafka-consumer-groups --describe --group g1 --members --verbose` is the other half of the diagnosis: its ASSIGNMENT column says who owns that partition. `kafka-consumer-groups --describe --group g1 --state` shows the group is still Stable, which is why nothing rebalanced under you.
Open at this step: step 4, Which member is it?Caught up again. t+9.400s
c2 started polling again and worked through its backlog a batch per poll. `kafka-consumer-groups --describe --group g1` has both partitions level, committed offset equal to log end offset. `kafka-get-offsets --topic orders` reads those same log ends straight off the brokers, without going through the group at all.
Open at this step: step 5, Caught up againWhat you start with
This scenario is played from the terminal (Ctrl and the backtick key). A two-partition topic, a group of two consumers, and one of them stops polling while the producer keeps appending. The lag table is the first thing an on-call engineer looks at, and it is enough to find the bad member without opening the inspector at all.
Step by step
- A group of two, nothing produced yet t+0.000s Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). Nothing has been produced yet, so nobody can be behind: `kafka-consumer-groups --list` shows the one group, and `kafka-consumer-groups --describe --group g1` already prints a row per partition. Neither row has a committed offset yet, so its LAG reads "-", not 0: in this table "nothing committed" and "nothing to catch up on" are different answers. Open at this step: step 1, A group of two, nothing produced yet
- Six records in, both members keeping up t+3.200s Both members are polling. `kafka-consumer-groups --describe --group g1` gives you one row per partition, each with a real CONSUMER-ID and a LAG at or near zero. A lone 1 there is not a problem: LAG is measured against the COMMITTED offset and auto-commit lands on the NEXT poll, so a record that has just been read still counts for one more tick. Open at this step: step 2, Six records in, both members keeping up
- One member stopped polling t+5.200s c2 got stuck in record processing while the producer kept appending. `kafka-consumer-groups --describe --group g1` now shows a "-" where one partition's consumer used to be, and that row's LAG pulling away from its sibling. Keep it on screen with `watch -n 1 kafka-consumer-groups --describe --group g1`: you get a dim header line carrying the command and the SIM clock, and the table under it rewritten in place as LAG climbs. The `-n 1` only decorates that header, so there is no wall clock to wait for. The re-render rides the displayed frames, so pausing the scenario freezes the header and the table with it. Open at this step: step 3, One member stopped polling
- Which member is it? t+6.800s The gap has widened. `kafka-consumer-groups --describe --group g1` shows it, but the lag table names the PARTITION, not the member, so `kafka-consumer-groups --describe --group g1 --members --verbose` is the other half of the diagnosis: its ASSIGNMENT column says who owns that partition. `kafka-consumer-groups --describe --group g1 --state` shows the group is still Stable, which is why nothing rebalanced under you. Open at this step: step 4, Which member is it?
- Caught up again t+9.400s c2 started polling again and worked through its backlog a batch per poll. `kafka-consumer-groups --describe --group g1` has both partitions level, committed offset equal to log end offset. `kafka-get-offsets --topic orders` reads those same log ends straight off the brokers, without going through the group at all. Open at this step: step 5, Caught up again
The takeaway
LAG is per partition, and it is measured against the COMMITTED offset. A stalled member still owns its partition but reads nothing, so the table shows a "-" consumer with its lag climbing while its sibling stays level - that one row is the diagnosis. Read the "-" carefully, though: this simulator stops crediting a member that is not reading, whereas real kafka-consumer-groups keeps printing a stuck-but-heartbeating member's CONSUMER-ID and prints "-" when a partition is UNOWNED. The climbing LAG next to a level sibling is the signal in both. Because the commit lands on the NEXT poll, a member that has just drained its partition can still read LAG 1 for one more poll. watch keeps the table on screen and re-renders on every displayed frame, so pausing the scenario freezes it too.
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=13.1.2" title="Kafka Simulator: Diagnose lag before opening the inspector" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/diagnose-lag/">Diagnose consumer lag with kafka-consumer-groups --describe</a></p>Further reading
Troubleshooting
More in Terminal ops
- Kafka from the terminal: hosts, ssh and the KRaft controller
- Kafka log segments on disk: leader vs follower
- Under-replicated vs under-min-ISR partitions in Kafka
- Reset Kafka consumer group offsets with --to-earliest
- Set retention.ms on a live topic with kafka-configs
- Kafka partition reassignment: execute, throttle, verify
- Graceful broker shutdown with kafka-server-stop
- Kafka tools on two clusters: --bootstrap-server decides