BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-25@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T223000Z
DTEND:20261105T231500Z
SUMMARY:Streamling: Lightweight\, Extensible Data Streaming Framework on Ap
 ache DataFusion
LOCATION:Room 2010
DESCRIPTION:Rafael Aguiar\, Yaroslav Tkachenko\n\nMost of the stream-proces
 sing systems are inherently complex. As an engineer\, you need to operate 
 a stateful distributed system even for simple scenarios. We built Streamli
 ng to explore what it takes to have a production streaming engine for appl
 ication developers\, not data platform engineers. It stays intentionally s
 imple\, with no distributed shuffle and no stateful joins\, and focuses on
  operational clarity\, extensibility\, and cloud-native deployment. It off
 ers tight integration with application backends and datastores (via HTTP\,
  Postgres and ClickHouse connectors). Streamling is a Rust-based streaming
  engine that uses Apache DataFusion planning and Apache Arrow RecordBatch 
 streams for real-time SQL/WASM transforms. Transitioning from row-based to
  column-based processing entailed many engineering trade-offs: handling ba
 ckpressure\, designing a checkpointing system\, and writing data efficient
 ly at the connector level. This talk is aimed at engineers building or eva
 luating streaming platforms\, as well as anyone exploring DataFusion beyon
 d batch analytics. Key takeaways include: DataFusion is proving to be a ve
 rsatile foundation for streaming\, not just batch. You don’t need distri
 buted stateful processing for many streaming workloads. Deliberately scopi
 ng down unlocks operational simplicity. Modern data streaming can go beyon
 d SQL: TypeScript transformations via WebAssembly\, Postgres as a streamin
 g sink and lookup source\, and HTTP endpoints for data enrichment.
URL:https://sf26.rmoff.net/s/streamling-lightweight-extensible-data-streami
 ng-framework-25
END:VEVENT
END:VCALENDAR