BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-81@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T193000Z
DTEND:20261105T201500Z
SUMMARY:No Data Engineers Required: DevEx and Operations for Thousands of S
 treaming Pipelines
LOCATION:Room 2004
DESCRIPTION:Xiao Meng\, Avishek Neupane\n\nA streaming platform has two sid
 es: application developers who want real-time data without becoming stream
 -processing experts\, and platform engineers who must keep thousands of te
 nant pipelines isolated\, observable\, and—ideally—boring to operate. 
 At Goldsky\, we built a multi-tenant declarative streaming platform twice:
  first as Mirror on Flink\, then as Turbo on Streamling\, our DataFusion-b
 ased engine. The engine changed\, but the goal did not: make streaming app
 lications easy for developers to build and maintain\, while keeping infras
 tructure and support manageable for a small platform team. This talk is a 
 production case study of what we learned from rebuilding both the engine a
 nd the control plane. On the DevEx side\, we built an end-to-end workflow 
 around the declarative YAML: pre-submit validation\, built-in OpenTelemetr
 y\, live inspect tooling to preview data as it flows through a pipeline\, 
 audit history of what changed between versions\, and clearer error ownersh
 ip—plus LLM-assisted root-cause hints and fix recommendations. Together\
 , these features turn streaming from a black box into something developers
  can review\, debug\, and ship with confidence. On the platform side\, we
 ’ll unpack the control-plane design that makes thousands of pipelines ma
 nageable: tenant isolation\, tiering\, secret management\, automated healt
 h checks \, safe rolling upgrades and lifecycle operations. The line that 
 mattered most was the control-plane/data-plane split. The control plane ow
 ns user intent; the data plane runs the streams. We tested that boundary t
 he hard way: a new engine and a new control plane. Migrating to the new st
 ack required almost no change to users' pipelines. What we learned is that
  DevEx and Operational Experience are one design problem. Good UX reduced 
 support load. Clean control-plane boundaries made user changes safer to sh
 ip. You’ll leave with a playbook for a declarative streaming platform th
 at stays self-service for developers and operable for the team running it
 —even as the engine underneath changes.
URL:https://sf26.rmoff.net/s/no-data-engineers-required-devex-and-operation
 s-for-81
END:VEVENT
END:VCALENDAR