BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-85@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T221000Z
DTEND:20261105T222500Z
SUMMARY:More pods won't save you: how Whatnot scaled Kafka producers past 5
 0\,000 pods
LOCATION:Innovation Theater
DESCRIPTION:Micah Denbraver\n\nYou have one Kafka topic. You have known con
 nection limits. You need to scale to 50\,000 pods. And you have one month.
  Where do you start? Whatnot runs a Python monolith that produces nearly a
 ll events to a single firehose Kafka topic. By late 2025\, load testing ha
 d identified a ceiling: producers timing out around 19\,000 pods\, brokers
  carrying 11\,000 client connections each against a recommended 5\,000\, c
 onnection cost scaling faster than throughput. The ceiling was known and i
 t wasn't urgent\, until early December\, when the team learned MrBeast was
  bringing a $1M giveaway that would peak at 583\,000+ concurrent viewers a
 nd require sustained operation at 50\,000+ backend pods. We had ten weeks.
  In 15 minutes\, we'll walk through the options that didn't work and why e
 ach failed for our architecture (adding brokers\, sidecars\, existing open
 -source proxies)\, the constraints that shaped the answer\, and how we end
 ed up with something simpler than every alternative on our shortlist which
  we built in a month\, and steady at 50\,000+ pods on event day. Audience 
 takeaways: a diagnostic pattern for connection-bound Kafka topologies\, wh
 y the obvious answers fail\, and a concrete reference for unblocking produ
 cer scale on a tight timeline.
URL:https://sf26.rmoff.net/s/more-pods-wont-save-you-how-whatnot-scaled-kaf
 ka-producers-85
END:VEVENT
END:VCALENDAR