From Streaming ETL to AI-Ready Data Pipelines: The Evolution of Flink CDC

The rise of AI applications has fundamentally changed what we expect from data infrastructure. Real-time features feed online inference models, lakehouse architectures (Iceberg, Paimon, Hudi) are becoming the de facto storage layer, and data teams are asked to keep thousands of operational tables in sync -- all while upstream schemas shift daily. Yet the data integration layer has not kept pace: per-table ETL pipelines become unmanageable at scale, schema changes break production jobs, and GUI-based tools are invisible to the AI agents that will increasingly operate our infrastructure. Apache Flink CDC was built to solve exactly these problems. In this session, we present its evolution from a CDC connector toolkit into a full-featured, AI-ready streaming data integration platform.

We start with YAML Pipeline API -- a declarative, zero-code approach to defining streaming ETL jobs (Source, Transform, Route, Sink) in a single file. Crucially, YAML is not just convenient for humans; it is a natural language interface between AI and infrastructure. LLMs can directly generate, review, and modify pipeline configurations as structured text -- something impossible with GUI-based tools. We demonstrate AI-assisted YAML generation and inline AI-powered transforms for model inference within the ETL pipeline itself.

We then deep dive Schema Evolution: Flink CDC automatically propagates upstream DDL changes (add/drop/rename columns, type changes, new tables) to downstream sinks without pipeline restarts. Five configurable behavior modes (evolve, try_evolve, lenient, ignore, exception) give operators fine-grained control over how schema changes flow through the pipeline.

Next, we dive into massive-scale lake ingestion: syncing 25,000+ tables from MySQL and PostgreSQL databases into Apache Iceberg, Paimon, or Hudi with a single Flink job. We share production metrics from real deployments -- sub-second end-to-end latency, 8-minute initialization (down from 45 minutes), automatic sharded-table merging, and over one year of stable operation. For the PostgreSQL + Iceberg audience, we demonstrate end-to-end schema evolution from PG DDL through to Iceberg table metadata.

Finally, we showcase the expanded connector ecosystem -- PostgreSQL, MySQL, and Fluss sources; Iceberg, Paimon, Hudi, and Fluss sinks; VARIANT type support for semi-structured data -- and discuss how Flink CDC integrates with Flink 2.x as a unified real-time data and AI platform.

Attendees will leave with actionable patterns for building production-grade, AI-ready streaming data pipelines, covering schema evolution, zero-code pipeline design, and massive-scale lake ingestion.