Scenario 01.2.5 · Rack-aware partitioning

Kafka rack-aware partitioning: partitioner.rack.aware

Producers & EOS · Advanced · about 3 min · Apache Kafka 4.4

Kafka 4.4 partitioner.rack.aware keeps keyless batches on partitions led in the producer's own zone, and falls back to every partition when that zone has no leader.

What you start with

Three brokers sit in three availability zones and each leads two of the six orders partitions. The producer runs in az-a (client.rack=az-a). At first its keyless batches visit every partition, so two batches in three cross a zone. Then partitioner.rack.aware, new in Kafka 4.4 (KIP-1123), keeps them on the partitions led in az-a, while a keyed record still hashes anywhere. When az-a loses its broker no leader is left in the zone, and the producer falls back to every partition until the leaders return.

Step by step

1. Producer in az-a t+0.000s Three brokers in three zones, each leading two of the six partitions: broker-1 (az-a) leads p0 and p3. The producer runs in az-a and sends keyless records. Open at this step: step 1, Producer in az-a
2. Every zone gets a batch t+1.600s partitioner.rack.aware is off (the default). The sticky partition moves p0, p1, p2, so the batches for p1 (az-b) and p2 (az-c) cross a zone. Open at this step: step 2, Every zone gets a batch
3. Rack-aware: az-a leaders only t+2.700s With partitioner.rack.aware=true the default partitioner picks each new sticky partition among the partitions led in client.rack, here p3 and p0. Kafka picks one at random; the simulator alternates so the rotation is visible. Open at this step: step 3, Rack-aware: az-a leaders only
4. A key still hashes t+3.400s Record K2 has a key, so it hashes to p2, led by broker-3 in az-c. Rack awareness only places keyless records, and the sticky partition did not move. Open at this step: step 4, A key still hashes
5. No leader left in az-a t+4.700s broker-1 is down and p0 and p3 elected leaders in az-b. No partition is led in the producer's zone any more. Open at this step: step 5, No leader left in az-a
6. Fallback: every partition t+5.600s The first batch stayed on p0, the sticky partition picked before the outage, even though its leader is now in az-b: the partitioner only re-picks when it switches. Those switches found no leader in az-a and fell back to every partition: p1, then p2. Open at this step: step 6, Fallback: every partition
7. Leaders back in az-a t+8.000s broker-1 restarted, caught up, and the preferred leader election gave it p0 and p3 back. The next switch picked from the az-a leaders again: p3, then p0. Open at this step: step 7, Leaders back in az-a

The takeaway

partitioner.rack.aware steers only the keyless records of the default partitioner: each new sticky partition comes from the partitions whose leader is in client.rack, or from all of them when there is none. The producer needs a client.rack (without one it fails to start with a ConfigException), and it helps only as far as leadership is spread across zones, so preferred leaders matter.

Configs in this scenario

• partitioner.rack.aware Since Kafka 4.4 (KIP-1123), whether the default partitioner picks each new sticky partition for keyless records among the partitions whose leader is in the producer's client.rack, falling back to every available partition when none is (default false); it needs a client.rack, or the producer fails to start with a ConfigException, and a custom partitioner ignores it.
• client.rack The rack (in the cloud, the availability zone) the client runs in (default empty); with a broker-side replica.selector.class such as RackAwareReplicaSelector a consumer fetches from a follower in its own rack instead of the leader, and since Kafka 4.4 a producer with partitioner.rack.aware=true sends keyless records to partitions whose leader is in its rack.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.2.5" title="Kafka Simulator: Rack-aware partitioning" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/rack-aware-partitioning/">Kafka rack-aware partitioning: partitioner.rack.aware</a></p>

Further reading

Apache Kafka documentation

More in Producers & EOS

• Kafka producer acks=0: fire and forget, silent data loss
• Kafka producer acks=1: acknowledged by the leader only
• Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR
• Kafka acks=1 data loss: min.insync.replicas does not help
• Kafka linger.ms vs batch.size: when the producer sends
• Kafka producer partitioners: keyed, round-robin and sticky
• Kafka compression.type: fewer, larger producer batches
• Kafka buffer.memory and max.block.ms: producer backpressure
• Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS
• Kafka enable.idempotence=false: a retry writes a duplicate
• Kafka idempotent producer dedup vs a plain producer
• Kafka max.in.flight.requests.per.connection and ordering

All scenarios