No Data Engineers Required: DevEx and Operations for Thousands of Streaming Pipelines

A streaming platform has two sides: application developers who want real-time data without becoming stream-processing experts, and platform engineers who must keep thousands of tenant pipelines isolated, observable, and—ideally—boring to operate.

At Goldsky, we built a multi-tenant declarative streaming platform twice: first as Mirror on Flink, then as Turbo on Streamling, our DataFusion-based engine. The engine changed, but the goal did not: make streaming applications easy for developers to build and maintain, while keeping infrastructure and support manageable for a small platform team.

This talk is a production case study of what we learned from rebuilding both the engine and the control plane.

On the DevEx side, we built an end-to-end workflow around the declarative YAML: pre-submit validation, built-in OpenTelemetry, live inspect tooling to preview data as it flows through a pipeline, audit history of what changed between versions, and clearer error ownership—plus LLM-assisted root-cause hints and fix recommendations. Together, these features turn streaming from a black box into something developers can review, debug, and ship with confidence.

On the platform side, we’ll unpack the control-plane design that makes thousands of pipelines manageable: tenant isolation, tiering, secret management, automated health checks , safe rolling upgrades and lifecycle operations.  The line that mattered most was the control-plane/data-plane split. The control plane owns user intent; the data plane runs the streams. We tested that boundary the hard way: a new engine and a new control plane. Migrating to the new stack required almost no change to users' pipelines.

What we learned is that DevEx and Operational Experience are one design problem. Good UX reduced support load. Clean control-plane boundaries made user changes safer to ship.

You’ll leave with a playbook for a declarative streaming platform that stays self-service for developers and operable for the team running it—even as the engine underneath changes.