Catching the Wave: PyFlink's missing DataFrame API

Every Python data engineer already knows how to write df.filter(...).group_by(...).agg(...). Give them that entry point into Flink and a whole community can build streaming and AI pipelines with the muscle memory they already have, while the Flink planner, runtime and checkpointing do the heavy lifting underneath.

That's the promise of FLIP-591, a new pyflink.dataframe module that adds a DataFrame API alongside the existing Table and DataStream APIs. In this lightning talk we'll show why it lives as its own dedicated API rather than reshaping the Table API, and why it borrows ergonomics from the modern generation of DataFrame libraries (Polars, Daft and Ray Data) rather than chasing pandas parity.

Then the streaming twist, where the DataFrame model is reimagined for an infinite source: lazy plans, changelog-aware iteration that yields RowKind, and bounded time-boxed previews so a notebook peek actually terminates. You'll leave knowing where PyFlink is heading and how to help shape it.