The Partition Tax: A Playbook for Right-Sizing Your Kafka Clusters

Internal research across sophisticated Kafka users suggests 40–70% of their cluster spend is wasted on excess partitions. These aren't beginners — Kafka simply makes over-provisioning the path of least resistance, and once a topic is keyed, you can't easily take partitions back.

This talk lays out a practical, layered strategy for getting that spend back, organized by who can act and when.

First, prevent: governance guardrails that stop new sprawl at provisioning time without turning the platform team into a ticket queue.

Then clean: use cost attribution and chargeback to show each team their own waste and hold them accountable to the cost.

Then fix the root cause: slow, IO-bound consumers are the most common reason teams reach for more partitions. Lots of partitions are not the answer. Share groups / queues for Kafka is not the answer. The real answer is parallel and concurrent consumption that preserves per-key ordering.

Finally, patch: what "lies" a Kafka proxy can tell to patch up infrastructure deficiencies, from using virtualization to consolidate non-prod and concentrating low-volume topics to an experimental approach to unblocking a slow consumer without touching its code.

You'll leave with a concrete checklist for auditing partition waste in your own clusters and a decision framework for which layer to reach for, mapped to your ordering requirements and your org's constraints.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.