Benchmarking Streaming Ingestion Engines for Apache Iceberg

Apache Iceberg is becoming the lakehouse's shared storage layer. Every major warehouse reads it, and a wave of new engines promise to stream Kafka into it. Once query engines share tables, the ingestion engine becomes a separate choice. It determines whether tables stay fresh, data stays correct, and files stay fast to query, goals that pull against each other. Warehouse benchmarks compare platforms with their own ingest and storage, leaving these engines without a common yardstick others can rerun and check. We introduce ingest-bench, built on three principles. Reproducible: every engine replays the same fixed workload under the same rules. Verifiable: the harness checks output tables directly, not engine metrics, and publishes the evidence. Extensible: new engines, workloads and test types plug in under the same scoring. We'll cover the design, what it showed about Apache Flink, Apache Spark and Eon Stream, our own engine, and how to apply it to your pipelines.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.