Streamling: Lightweight, Extensible Data Streaming Framework on Apache DataFusion

Most of the stream-processing systems are inherently complex. As an engineer, you need to operate a stateful distributed system even for simple scenarios.

We built Streamling to explore what it takes to have a production streaming engine for application developers, not data platform engineers. It stays intentionally simple, with no distributed shuffle and no stateful joins, and focuses on operational clarity, extensibility, and cloud-native deployment. It offers tight integration with application backends and datastores (via HTTP, Postgres and ClickHouse connectors).

Streamling is a Rust-based streaming engine that uses Apache DataFusion planning and Apache Arrow RecordBatch streams for real-time SQL/WASM transforms. Transitioning from row-based to column-based processing entailed many engineering trade-offs: handling backpressure, designing a checkpointing system, and writing data efficiently at the connector level.

This talk is aimed at engineers building or evaluating streaming platforms, as well as anyone exploring DataFusion beyond batch analytics. Key takeaways include:

• DataFusion is proving to be a versatile foundation for streaming, not just batch.
• You don’t need distributed stateful processing for many streaming workloads. Deliberately scoping down unlocks operational simplicity.
• Modern data streaming can go beyond SQL: TypeScript transformations via WebAssembly, Postgres as a streaming sink and lookup source, and HTTP endpoints for data enrichment.