Scenario 11.0.2 · Cooperative consumption beyond partition count

More Kafka consumers than partitions with share groups

Share groups · Intermediate · about 2 min · Apache Kafka 4.3

THREE share-group members on ONE partition - records load-balance across all three (share groups decouple consumption from partition count, KIP-932).

What you start with

In a classic consumer group one partition can feed at most one consumer, so adding members beyond the partition count leaves them idle. A share group has no partition assignment: this scenario puts THREE members on a SINGLE partition and, with max.poll.records=1, each poll hands one record to each member - fanning six jobs out across all three.

Step by step

1. Cluster ready · idle t+0.000s One partition, one share group with THREE members (w1, w2, w3). A classic consumer group could keep only one of them busy on a single partition. Press ▶. Open at this step: step 1, Cluster ready · idle
2. Six jobs queued t+0.500s The producer wrote six jobs (HW=6); they are durable and waiting. The share group has not polled yet - every record is still 'available'. Open at this step: step 2, Six jobs queued
3. First poll · three members each grab one t+0.600s No partition assignment: all three members poll the same partition and acquire DISJOINT offsets - w1→0, w2→1, w3→2 (max.poll.records=1). Three consumers work in parallel on one partition. Open at this step: step 3, First poll · three members each grab one
4. Second poll · first batch acked, next three acquired t+1.200s Offsets 0/1/2 auto-accept (→ acked) a beat after acquisition, then w1/w2/w3 acquire the next three - 3/4/5. The queue drains across members, not across partitions. Open at this step: step 4, Second poll · first batch acked, next three acquired
5. Queue drained t+1.800s Offsets 3/4/5 are now acked too - all six jobs processed, three members sharing one partition. Consumption scaled past the partition count (KIP-932). Open at this step: step 5, Queue drained

The takeaway

Share groups let you scale consumers beyond the partition count - add members to drain a backlog faster without ever repartitioning the topic.

Configs in this scenario

• max.poll.records The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=11.0.2" title="Kafka Simulator: Cooperative consumption beyond partition count" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/cooperative-beyond-partitions/">More Kafka consumers than partitions with share groups</a></p>

Further reading

KIP-932 on the Apache Kafka wiki

More in Share groups

• Kafka share groups: queue semantics with KIP-932
• Kafka share group acquisition lock timeout and redelivery
• Kafka share consumer acknowledge: accept, release, reject
• Kafka share group poison pill: delivery limit and archiving
• Kafka share.acknowledgement.mode: implicit vs explicit
• Kafka share.acquire.mode: batch_optimized vs record_limit
• Renew a Kafka share group acquisition lock (KIP-1222)
• Kafka share group lag: SPSO, SPEO and the acked prefix

All scenarios