Keeping the Balance with Kafka Streams

Apache Kafka’s scalability and resilience depend not just on its brokers, but crucially on its client ecosystem—most notably, Kafka Streams. Maintaining stability forces Streams applications to redistribute partitioned workloads during infrastructure shifts. These rebalances are notorious for causing operational friction, performance degradation, and unexpected downtime.

KIP-1071 (Streams groups) introduces a major evolutionary step towards resolving this friction. Built on the foundation of next-generation consumer groups (KIP-848), Streams groups move task assignment to the broker side and enable fully incremental rebalancing. The result is faster, more stable, and predictable coordination that fundamentally changes how Kafka Streams handles scaling in, scaling out, and failing over.

In this talk, we will dive into the core assignment logic, specifically focusing on the performance of the new protocol, its sticky task assignor, and a direct comparison to classic rebalancing. We will also explore a completely new incarnation of the high-availability assignor, which is designed explicitly to allow smooth scaling out for state-heavy applications. You will see how this updated design provides a much cleaner architecture by decoupling warmup tasks from the actual assignment logic and splitting the process into distinct assignment and reconciliation phases.

Join this session to learn about the latest advancements in Streams groups and gain the insights needed to migrate your production applications with confidence. You will walk away with a clear understanding of how these new protocols minimize operational headaches and keep your stream processing pipelines running smoothly.