BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-49@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261104T220000Z
DTEND:20261104T224500Z
SUMMARY:Handling Surges in Petabyte-Scale Streaming Systems by Doing Nothin
 g
LOCATION:Room 2002
DESCRIPTION:Jeff Xiang\n\nWhen streaming data at petabyte scale\, one of th
 e most painful on-call scenarios is handling sudden traffic surges that ov
 erload servers\, trigger cascading failures\, and wipe out service availab
 ility across a large blast radius. At modern throughput levels\, scaling o
 perations are simply not fast enough to prevent unexpected 10–20x spikes
  from taking down dozens of streaming pipelines and their neighbors. The c
 lassical mitigation is to overprovision\, add proactive alerting\, and sim
 ply “react quickly”. In this talk\, we present a TCP-based congestion 
 control approach that tackles the problem at its root and eliminates the n
 eed for manual on-call intervention. At Pinterest\, we have productionized
  this TCP-based flow control solution in a 50 GB/s streaming system that p
 owers machine learning across the company. By setting up the appropriate e
 nd-to-end flow control mechanisms\, we guard against sudden surges of any 
 magnitude by propagating backpressure gracefully\, predictably\, and fully
  autonomously. We will walk through the key concepts in networking\, memor
 y management\, and backpressure that matter in large-scale streaming syste
 ms\, and then unpack the exact mechanism we built to solve this problem. T
 he audience will leave with a set of production-ready ideas and patterns t
 hat can be replicated in their own streaming environments that are far mor
 e cost-efficient and operationally lightweight than classical solutions. B
 eyond eliminating the catastrophic risk of sudden traffic surges\, we will
  share concrete and replicable takeaways from running these concepts in pr
 oduction at scale\, including: Designing streaming topologies that rely on
  backpressure instead of excess capacity Safely transforming scaling and l
 oad-balancing into reactive operations\, reducing unnecessary early alerti
 ng and interventions Simplifying capacity planning for organic growth Lowe
 ring infrastructure cost by running denser workloads with minimal buffer h
 eadroom
URL:https://sf26.rmoff.net/s/handling-surges-in-petabyte-scale-streaming-sy
 stems-by-doing-49
END:VEVENT
END:VCALENDAR