The Producer Memory Math That Breaks at Scale (KIP-1332)

Here's something a lot of people don't realize about the Kafka producer: its memory usage scales with the number of partitions you're actively writing to, not with the rate of data actually flowing through. Behind the scenes, the producer sets aside a fixed chunk of memory for every partition it's actively writing to. So your real memory usage isn't "bytes per second" — it's "number of busy partitions × a fixed batch size."

For a long time this rarely came up. But more and more teams are moving to higher-latency, object-storage-backed setups — like the diskless topics in KIP-1150 — where reaching the same throughput means using a larger batch size. Combine a larger batch size with a topic that spreads writes across many partitions, and the per-partition math adds up: you tend to end up either limiting throughput or over-provisioning memory for a worst case that's hard to predict.

Let's talk about what KIP-1332 brings to the table to address this — a more flexible way to allocate producer memory that scales with your actual traffic rather than your partition count, and what that means for how you size and tune producers going forward.

Who is it for

• Application developers running into producer memory pressure, throughput limits, or unpredictable memory usage

• Platform and infrastructure engineers sizing and tuning Kafka producers

• Teams with high-partition topics, custom partitioners, or keyed high-cardinality workloads

• Anyone exploring higher-latency or object-storage-backed Kafka setups (e.g. diskless topics) where larger batch sizes come into play