Agenda

EventBus at Dropbox: Self-Serve Millions of Events Per Second with Near-Zero Ops

David Yun, Dropbox

Thu, Nov 5, 10:30⁠–⁠11:15 AMRoom 2010

Kafka OperationsAdvancedBreakout session

When operating 6 platforms with a team of 3 and actively developing the 7th, near-zero ops is no longer a luxury: it is a necessity.

In this talk, we deep dive into Dropbox's standard event processing platform: EventBus. Today EventBus processes hundreds of billions of events per day across 50+ use cases, serving as the backbone for data pipelines, search indexing, and AI ingestion company-wide. The platform abstracts away the complexities of operating Kafka consumers and transforms Kafka consumption into familiar stateless remote procedure calls focusing on business logic. Users of the platform are able to self-onboard use cases that consume millions of messages per second, and everyone still gets to sleep at night. EventBus carries the golden standard in platform reliability at Dropbox, with 0 SEVs incurred since GA >1 year ago despite having 0 resources dedicated for KTLO.

This talk covers advanced topics such as going beyond partition-level parallelism while retaining message key ordering, how we deal with hot partitions, protecting tenants from noisy neighbor issues, and how we are able to operate cost effectively on top of being operationally light.

Whether you already run Kafka at scale or are weighing whether to build a platform layer over it, you will leave with a concrete blueprint for turning per-team consumer sprawl into a paved road engineers actually want to use.

Speaker

Also on at this time

Add to calendar

Google Calendar Outlook.com Outlook (work or school) Apple Calendar (.ics)