BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-167@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T223000Z
DTEND:20261105T231500Z
SUMMARY:Forget Everything You Know About Ingestion Engines
LOCATION:Room 2000
DESCRIPTION:Doron Porat\, Liran Yogev\n\nWe land 200+ billion events a day\
 , around 40\,000 Avro files per minute\, into Apache Iceberg\, queryable i
 n under a minute. Our first production system did this on Flink with Paimo
 n. It worked\, but at this fan-out and freshness target we kept paying for
  the same things: tiny files flooding the table\, compaction and stats run
 ning as after-the-fact jobs that re-read the data\, one hot partition pinn
 ing a single task slot\, and elasticity gated by checkpoint and state redi
 stribution. None of these are bugs. They are the honest cost of a general-
 purpose\, keyed-state engine pointed at a problem that is really just file
  fan-in to a table. So we rebuilt it in Rust as a set of independent\, sin
 gle-purpose services: writers that just flush Parquet\, compactors that ju
 st merge and sort\, and one committer that is the only component touching 
 Iceberg metadata. Nothing shares state\, each scales on its own signal\, a
 nd losing a worker only re-delivers its own in-flight files. Owning the wh
 ole path also let us fold in things a bolt-on engine cannot: inline statis
 tics computed in the same write pass\, clustering and LSM-style compaction
  on ingest\, dedup\, and read-free metadata commits. This is a blueprint t
 alk with the trade-offs left in. We will walk the full architecture\, how 
 the roles decouple\, how correctness holds without a global checkpoint\, w
 here it cost us\, and where it should not be used. You will leave with a c
 oncrete design you can reason about for your own ingestion workloads\, and
  a sharper sense of when to reach for a purpose-built path versus a genera
 l engine. By attending this session\, your contact information may be shar
 ed with the sponsor for relevant follow up for this event only.
URL:https://sf26.rmoff.net/s/forget-everything-you-know-about-ingestion-eng
 ines-167
END:VEVENT
END:VCALENDAR