Surviving the Surge: Netflix's playbook for Kafka Resilience

How would your Kafka cluster fare against a 10x increase in traffic? More critically, could you guarantee sustained high throughput under an extreme 100x load? What systems must be in place to meet such a challenge? At Netflix, rapidly expanding live streaming events place unprecedented demands on our core Kafka infrastructure, forcing us to address these critical questions. We're excited to share our solutions with you in this talk!

While Kafka is fundamentally stable, ensuring continuous service availability during extreme and often unpredictable overload scenarios demands more than out-of-the-box capabilities. This talk presents strategies to augment Kafka deployments, ensuring they are more resilient to unexpected overload and equipped with mechanisms to proactively manage and mitigate such situations, thereby maximizing operational uptime and minimizing data loss or service degradation.

Our presentation will cover critical improvements for the broker and client. We present an approach to guarantee broker stability during overload. We also introduce an approach that transforms clients from naive data producers/consumers into active participants who maintain cluster stability by dynamically adapting their behavior based on real-time load conditions.

Join us for a deep dive into Netflix’s techniques for Kafka stability and peak throughput when pushed to its limits. We will discuss insights and key takeaways that may benefit your own Kafka strategy.