BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-132@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261104T231500Z
DTEND:20261105T000000Z
SUMMARY:From Streaming ETL to AI-Ready Data Pipelines: The Evolution of Fli
 nk CDC
LOCATION:Room 2010
DESCRIPTION:Leonard Xu\n\nThe rise of AI applications has fundamentally cha
 nged what we expect from data infrastructure. Real-time features feed onli
 ne inference models\, lakehouse architectures (Iceberg\, Paimon\, Hudi) ar
 e becoming the de facto storage layer\, and data teams are asked to keep t
 housands of operational tables in sync -- all while upstream schemas shift
  daily. Yet the data integration layer has not kept pace: per-table ETL pi
 pelines become unmanageable at scale\, schema changes break production job
 s\, and GUI-based tools are invisible to the AI agents that will increasin
 gly operate our infrastructure. Apache Flink CDC was built to solve exactl
 y these problems. In this session\, we present its evolution from a CDC co
 nnector toolkit into a full-featured\, AI-ready streaming data integration
  platform. We start with YAML Pipeline API -- a declarative\, zero-code ap
 proach to defining streaming ETL jobs (Source\, Transform\, Route\, Sink) 
 in a single file. Crucially\, YAML is not just convenient for humans; it i
 s a natural language interface between AI and infrastructure. LLMs can dir
 ectly generate\, review\, and modify pipeline configurations as structured
  text -- something impossible with GUI-based tools. We demonstrate AI-assi
 sted YAML generation and inline AI-powered transforms for model inference 
 within the ETL pipeline itself. We then deep dive Schema Evolution: Flink 
 CDC automatically propagates upstream DDL changes (add/drop/rename columns
 \, type changes\, new tables) to downstream sinks without pipeline restart
 s. Five configurable behavior modes (evolve\, try_evolve\, lenient\, ignor
 e\, exception) give operators fine-grained control over how schema changes
  flow through the pipeline. Next\, we dive into massive-scale lake ingesti
 on: syncing 25\,000+ tables from MySQL and PostgreSQL databases into Apach
 e Iceberg\, Paimon\, or Hudi with a single Flink job. We share production 
 metrics from real deployments -- sub-second end-to-end latency\, 8-minute 
 initialization (down from 45 minutes)\, automatic sharded-table merging\, 
 and over one year of stable operation. For the PostgreSQL + Iceberg audien
 ce\, we demonstrate end-to-end schema evolution from PG DDL through to Ice
 berg table metadata. Finally\, we showcase the expanded connector ecosyste
 m -- PostgreSQL\, MySQL\, and Fluss sources; Iceberg\, Paimon\, Hudi\, and
  Fluss sinks; VARIANT type support for semi-structured data -- and discuss
  how Flink CDC integrates with Flink 2.x as a unified real-time data and A
 I platform. Attendees will leave with actionable patterns for building pro
 duction-grade\, AI-ready streaming data pipelines\, covering schema evolut
 ion\, zero-code pipeline design\, and massive-scale lake ingestion.
URL:https://sf26.rmoff.net/s/from-streaming-etl-to-ai-ready-data-pipelines-
 the-evolution-132
END:VEVENT
END:VCALENDAR