Scenario 00.2.3 · Log start offset

Kafka log start offset: why the earliest offset is not 0

Anatomy · Intermediate · about 2 min · Apache Kafka 4.3

Offsets need not start at 0: retention deletes the oldest records, advancing the log start offset - the earliest readable record.

What you start with

Offsets don’t always begin at 0. Records are written in two batches, then a time-based retention sweep (retention.ms) deletes the older batch - advancing the log start offset so the earliest readable record is no longer offset 0.

Step by step

1. Empty log t+0.000s A topic with retention.ms set. Records will be written in two batches. Open at this step: step 1, Empty log
2. Offsets 0-4 · log start offset = 0 t+1.400s Five records occupy offsets 0-4. The log start offset is 0 - offset 0 is the earliest readable record. Open at this step: step 2, Offsets 0-4 · log start offset = 0
3. Old batch aged out · log start offset = 3 t+2.100s The retention sweep deleted the older batch (offsets 0-2) once it aged past retention.ms. Real Kafka deletes whole log segments, so this assumes offsets 0-2 sit in their own closed segment. The log start offset advanced to 3 - now the earliest readable record. Offsets 0-2 are gone forever; a consumer reading from 'earliest' begins at 3. Open at this step: step 3, Old batch aged out · log start offset = 3

The takeaway

The log start offset marks the oldest record still retained; retention deletes aged segments from the head of the log (whole segments, not single records), so consumers must tolerate an earliest offset greater than 0.

Configs in this scenario

• retention.ms How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=00.2.3" title="Kafka Simulator: Log start offset" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/log-start-offset/">Kafka log start offset: why the earliest offset is not 0</a></p>

Further reading

Apache Kafka documentation

More in Anatomy

• Kafka topics, partitions and offsets explained
• Kafka write path: one record from producer to consumer
• Kafka message keys: how a key picks the partition
• Kafka offsets are per partition, not global
• Kafka replication factor, leader and follower replicas
• Kafka high watermark vs log end offset (LEO)
• Why Kafka consumers cannot read past the high watermark

All scenarios