BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-135@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T193000Z
DTEND:20261105T201500Z
SUMMARY:Lessons from migrating Flink Streaming and Batch platform to Kubern
 etes at LinkedIn
LOCATION:Room 2002
DESCRIPTION:Sanil Jain\n\nIn this talk we share the design decisions and le
 ssons learned migrating LinkedIn's data processing platform — both strea
 ming and batch on Flink — to Kubernetes replacing YARN federations of 20
 \,000+ nodes. Same engine; different answers along almost every axis Deplo
 yment pattern: Operators pay off when jobs live forever\, not hours. Adopt
 ing Flink Kubernetes Operator from open source for Streaming and K8s nativ
 e submission for batch. Quota management: Streaming pays for guarantees; b
 atch pays for elasticity; mixing the two burns you. Streaming gets dedicat
 ed node pools; batch gets org-level quotas scheduled by Volcano (queue/pri
 ority/gang). Disruption readiness: Same goal\, opposite mechanism: protect
  the state where it lives. Insights into developing custom disruption hand
 lers for Streaming Jobs. Building Support for shuffle service Apache Celeb
 orn for Flink to ensure — a transient TM loss doesn't recompute upstream
  stages. Cluster federation. A logical-cluster abstraction over multiple p
 hysical K8s clusters (20k+ nodes) - how we route batch jobs across physica
 l clusters transparently\, while streaming pools stay per-fabric for guara
 ntees. Containerization.: A look at the layered-image and localization imp
 rovements that took deployment from minutes to seconds\, making Flink fram
 ework rollouts a zero-touch event. State handling. Insights into the state
 -locality choices that determine whether a TM restart costs seconds or min
 utes Identity\, secrets\, and observability. Insights into identity/secret
  management and the unified observability stack we built to serve streamin
 g and batch from a single platform Migration strategy. Two parallel migrat
 ions — thousands of streaming & batch jobs. Lessons on migrating at scal
 e without big-bang risk. Takeaway. Whether you're greenfielding a Flink pl
 atform or converging two existing ones\, the right answers along every axi
 s turn on workload lifetime and state. We'll share the decisions that held
  up\, the ones we reversed\, and a checklist others can apply on day one.
URL:https://sf26.rmoff.net/s/lessons-from-migrating-flink-streaming-and-bat
 ch-platform-to-135
END:VEVENT
END:VCALENDAR