BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-143@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T223000Z
DTEND:20261105T231500Z
SUMMARY:What Would It Take to Shrink a Kafka Partition? A Portable Pattern 
 for Live Repartitioning — and Where Each Broker Hits Its Limit
LOCATION:Room 2004
DESCRIPTION:Manideep Bora\n\nDynamic repartitioning — changing the partit
 ion count of a keyed stream while it's in use — is a problem every strea
 ming platform punts on\, and it surfaces wherever keyed work is sharded ac
 ross workers under changing load. Kafka only adds partitions\, never remov
 es them (KIP-253 has sat unimplemented since 2018) — a principled conseq
 uence of its offset-and-log model\, not an oversight. Kinesis reshards one
  direction per operation; Solace documents scaling as service-affecting wi
 th possible loss. The recurring tax is the same: a maintenance window\, lo
 st messages\, broken ordering\, or tight coordinator-to-producer coupling.
  This session argues the right unit of solution is a portable coordination
  pattern\, not another broker feature — because the problem is fundament
 ally about who reroutes keys\, when\, and how per-key order survives the s
 witch. The pattern is Buffer–Drain–Replay (BDR): it rescales a stream 
 up or down while preserving per-key ordering\, never dropping an acknowled
 ged message\, and never pausing producer ingestion. BDR is defined over an
  abstract substrate by five primitives — observable out-of-band signalin
 g\, a durable append-ordered buffer\, dedup-by-id\, a dynamic consumer lif
 ecycle\, and application-owned bidirectional routing — coordinated throu
 gh a config signal so producers never call the coordinator and can run in 
 any language. Correctness reduces to a single invariant\, replay-before-di
 rect\, backed by crash-resumable\, CAS-checkpointed execution and TTL-base
 d leader election. I'll demonstrate BDR live on a local NATS JetStream set
 up with a visual UI that makes the buffer→drain→replay phases and the 
 per-key ordering check observable in real time\, and present measured resu
 lts from a distributed\, multi-AZ Amazon EKS deployment (256 keys\, ~6\,00
 0 msg/s) — including a same-substrate baseline that quantifies what a na
 ive count-switch and a stop-the-world repartition actually cost. The work 
 comes out of running per-key-ordered streams in production at a trading pl
 atform\, where scaling during peak load can't mean halting transactions or
  reordering a single account's events. Attendees leave with three things: 
 a concrete protocol they can implement on any conforming system; a five-pr
 imitive checklist for evaluating their own broker — Kafka\, Pulsar\, Kin
 esis\, Solace\, or Pravega — against the pattern\, including exactly whe
 re each is constrained; and a cautionary measurement story. We first credi
 ted the ordering guarantee to a latency margin; direct measurement violate
 d that margin by two orders of magnitude while ordering held perfectly —
  proving the guarantee is structural\, not temporal. Zero violations\, zer
 o loss\, ingestion sustained through every repartition\, consumers resumin
 g in 155 ms.
URL:https://sf26.rmoff.net/s/what-would-it-take-to-shrink-a-kafka-partition
 -a-portable-143
END:VEVENT
END:VCALENDAR