Forget Everything You Know About Ingestion Engines

We land 200+ billion events a day, around 40,000 Avro files per minute, into Apache Iceberg, queryable in under a minute. Our first production system did this on Flink with Paimon. It worked, but at this fan-out and freshness target we kept paying for the same things: tiny files flooding the table, compaction and stats running as after-the-fact jobs that re-read the data, one hot partition pinning a single task slot, and elasticity gated by checkpoint and state redistribution. None of these are bugs. They are the honest cost of a general-purpose, keyed-state engine pointed at a problem that is really just file fan-in to a table. So we rebuilt it in Rust as a set of independent, single-purpose services: writers that just flush Parquet, compactors that just merge and sort, and one committer that is the only component touching Iceberg metadata. Nothing shares state, each scales on its own signal, and losing a worker only re-delivers its own in-flight files. Owning the whole path also let us fold in things a bolt-on engine cannot: inline statistics computed in the same write pass, clustering and LSM-style compaction on ingest, dedup, and read-free metadata commits. This is a blueprint talk with the trade-offs left in. We will walk the full architecture, how the roles decouple, how correctness holds without a global checkpoint, where it cost us, and where it should not be used. You will leave with a concrete design you can reason about for your own ingestion workloads, and a sharper sense of when to reach for a purpose-built path versus a general engine.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.