Monedula

Scenario 11.0.9 · SPSO/SPEO/lag: the share-group state machine

Kafka share group lag: SPSO, SPEO and the acked prefix

Share groups · Advanced · about 3 min · Apache Kafka 4.3

Share-Partition-Start-Offset advances only over the CONTIGUOUS acked prefix; SPEO is the high watermark; lag = SPEO − SPSO (KIP-932/KIP-1226).

Run this scenario in the playground

Acked above a gap - SPSO stuck. t+1.000s

The key lesson. We acked 0,1 (SPSO advanced to 2) then 3,4 - but deliberately NOT 2. Even though 3 and 4 are acked, SPSO stays at 2: it only advances over the CONTIGUOUS acked prefix, so an acked record above an in-flight gap stays inside the lag window (lag 3, not 1).

Open at this step: step 4, Acked above a gap - SPSO stuck

What you start with

A share partition tracks three offsets (KIP-1226): SPEO (the end of available records, i.e. the high watermark), SPSO (the first offset past the CONTIGUOUS acked prefix), and lag = SPEO − SPSO. With five records and explicit acks we accept 0,1 then 3,4 but leave the GAP at offset 2 - so SPSO stays at 2 until offset 2 is finally acked and the whole run collapses, SPSO leaping to 5 (lag 0).

Step by step

  1. Cluster ready · idle t+0.000s One explicit-ack share group, one member, an empty partition. SPSO = SPEO = 0, lag 0. We'll watch the three KIP-1226 share-partition offsets move as records flow through the available → acquired → acked state machine. Press ▶. Open at this step: step 1, Cluster ready · idle
  2. Five records produced · backlog t+0.200s Five records are produced and committed → SPEO (the Share-Partition-End-Offset, equal to the high watermark) jumps to 5. None processed yet, so SPSO is still 0 and lag = SPEO − SPSO = 5: a full backlog, even before the group has polled. Open at this step: step 2, Five records produced · backlog
  3. All five acquired t+0.700s First poll (600ms) acquires all five (deliveryCount 1). They are in-flight, not terminal - acquiring is a lock, not a commit - so SPSO does NOT move and lag is still 5. Open at this step: step 3, All five acquired
  4. Acked above a gap - SPSO stuck t+1.000s The key lesson. We acked 0,1 (SPSO advanced to 2) then 3,4 - but deliberately NOT 2. Even though 3 and 4 are acked, SPSO stays at 2: it only advances over the CONTIGUOUS acked prefix, so an acked record above an in-flight gap stays inside the lag window (lag 3, not 1). Open at this step: step 4, Acked above a gap - SPSO stuck
  5. Gap filled - SPSO leaps t+1.200s Acking offset 2 fills the gap - the whole run 0..4 is now contiguous-acked, so SPSO leaps straight to 5 and lag collapses to 0. SPSO is the durable 'everything below here is done' marker the share-group coordinator persists. Open at this step: step 5, Gap filled - SPSO leaps

The takeaway

SPSO only advances over a contiguous acked run, so a single un-acked record in the middle pins the start offset - just as a consumer group’s committed offset cannot move past an uncommitted gap.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=11.0.9" title="Kafka Simulator: SPSO/SPEO/lag: the share-group state machine" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/spso-speo-lag/">Kafka share group lag: SPSO, SPEO and the acked prefix</a></p>

Further reading

KIP-1226 on the Apache Kafka wiki

More in Share groups

All scenarios