Lessons from migrating Flink Streaming and Batch platform to Kubernetes at LinkedIn

In this talk we share the design decisions and lessons learned migrating LinkedIn's data processing platform — both streaming and batch on Flink — to Kubernetes replacing YARN federations of 20,000+ nodes. Same engine; different answers along almost every axis

• Deployment pattern: Operators pay off when jobs live forever, not hours. Adopting Flink Kubernetes Operator from open source for Streaming and K8s native submission for batch.
• Quota management: Streaming pays for guarantees; batch pays for elasticity; mixing the two burns you. Streaming gets dedicated node pools; batch gets org-level quotas scheduled by Volcano (queue/priority/gang).
• Disruption readiness: Same goal, opposite mechanism: protect the state where it lives. Insights into developing custom disruption handlers for Streaming Jobs. Building Support for shuffle service Apache Celeborn for Flink to ensure — a transient TM loss doesn't recompute upstream stages. 
• Cluster federation. A logical-cluster abstraction over multiple physical K8s clusters (20k+ nodes) - how we route batch jobs across physical clusters transparently, while streaming pools stay per-fabric for guarantees.
• Containerization.: A look at the layered-image and localization improvements that took deployment from minutes to seconds, making Flink framework rollouts a zero-touch event.

• State handling. Insights into the state-locality choices that determine whether a TM restart costs seconds or minutes

• Identity, secrets, and observability. Insights into identity/secret management and the unified observability stack we built to serve streaming and batch from a single platform
• Migration strategy. Two parallel migrations — thousands of streaming & batch jobs. Lessons on migrating at scale without big-bang risk.

Takeaway. Whether you're greenfielding a Flink platform or converging two existing ones, the right answers along every axis turn on workload lifetime and state. We'll share the decisions that held up, the ones we reversed, and a checklist others can apply on day one.