BEGIN:VCALENDAR VERSION:2.0 PRODID:-//sf26-agenda//EN CALSCALE:GREGORIAN METHOD:PUBLISH BEGIN:VEVENT UID:SESS-103@sf26-agenda DTSTAMP:20261020T151709Z DTSTART:20261105T181000Z DTEND:20261105T182500Z SUMMARY:The Producer Memory Math That Breaks at Scale (KIP-1332) LOCATION:Innovation Theater DESCRIPTION:Nikita Shupletsov\n\nHere's something a lot of people don't rea lize about the Kafka producer: its memory usage scales with the number of partitions you're actively writing to\, not with the rate of data actually flowing through. Behind the scenes\, the producer sets aside a fixed chun k of memory for every partition it's actively writing to. So your real mem ory usage isn't "bytes per second" — it's "number of busy partitions × a fixed batch size." For a long time this rarely came up. But more and mor e teams are moving to higher-latency\, object-storage-backed setups — li ke the diskless topics in KIP-1150 — where reaching the same throughput means using a larger batch size. Combine a larger batch size with a topic that spreads writes across many partitions\, and the per-partition math ad ds up: you tend to end up either limiting throughput or over-provisioning memory for a worst case that's hard to predict. Let's talk about what KIP- 1332 brings to the table to address this — a more flexible way to alloca te producer memory that scales with your actual traffic rather than your p artition count\, and what that means for how you size and tune producers g oing forward. Who is it for Application developers running into producer m emory pressure\, throughput limits\, or unpredictable memory usage Platfor m and infrastructure engineers sizing and tuning Kafka producers Teams wit h high-partition topics\, custom partitioners\, or keyed high-cardinality workloads Anyone exploring higher-latency or object-storage-backed Kafka s etups (e.g. diskless topics) where larger batch sizes come into play URL:https://sf26.rmoff.net/s/the-producer-memory-math-that-breaks-at-scale- kip-1332-103 END:VEVENT END:VCALENDAR