What Would It Take to Shrink a Kafka Partition? A Portable Pattern for Live Repartitioning — and Where Each Broker Hits Its Limit

Dynamic repartitioning — changing the partition count of a keyed stream while it's in use — is a problem every streaming platform punts on, and it surfaces wherever keyed work is sharded across workers under changing load. Kafka only adds partitions, never removes them (KIP-253 has sat unimplemented since 2018) — a principled consequence of its offset-and-log model, not an oversight. Kinesis reshards one direction per operation; Solace documents scaling as service-affecting with possible loss. The recurring tax is the same: a maintenance window, lost messages, broken ordering, or tight coordinator-to-producer coupling. This session argues the right unit of solution is a portable coordination pattern, not another broker feature — because the problem is fundamentally about who reroutes keys, when, and how per-key order survives the switch.

The pattern is Buffer–Drain–Replay (BDR): it rescales a stream up or down while preserving per-key ordering, never dropping an acknowledged message, and never pausing producer ingestion. BDR is defined over an abstract substrate by five primitives — observable out-of-band signaling, a durable append-ordered buffer, dedup-by-id, a dynamic consumer lifecycle, and application-owned bidirectional routing — coordinated through a config signal so producers never call the coordinator and can run in any language. Correctness reduces to a single invariant, replay-before-direct, backed by crash-resumable, CAS-checkpointed execution and TTL-based leader election. I'll demonstrate BDR live on a local NATS JetStream setup with a visual UI that makes the buffer→drain→replay phases and the per-key ordering check observable in real time, and present measured results from a distributed, multi-AZ Amazon EKS deployment (256 keys, ~6,000 msg/s) — including a same-substrate baseline that quantifies what a naive count-switch and a stop-the-world repartition actually cost.

The work comes out of running per-key-ordered streams in production at a trading platform, where scaling during peak load can't mean halting transactions or reordering a single account's events. Attendees leave with three things: a concrete protocol they can implement on any conforming system; a five-primitive checklist for evaluating their own broker — Kafka, Pulsar, Kinesis, Solace, or Pravega — against the pattern, including exactly where each is constrained; and a cautionary measurement story. We first credited the ordering guarantee to a latency margin; direct measurement violated that margin by two orders of magnitude while ordering held perfectly — proving the guarantee is structural, not temporal. Zero violations, zero loss, ingestion sustained through every repartition, consumers resuming in 155 ms.