BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-59@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261104T220000Z
DTEND:20261104T224500Z
SUMMARY:OpenLineage To Track Them All: From Kafka To The LakeHouse
LOCATION:Room 2010
DESCRIPTION:Karel Alfonso Sague\, Chad Harris\n\nMost teams can draw their 
 data architecture on a whiteboard but can't answer a basic governance ques
 tion: when a value in a dataset looks wrong\, where did it come from\, and
  what downstream tables and models does it affect? In a stack spanning Kaf
 ka Connect\, Flink\, Spark\, and Iceberg\, lineage breaks at every engine 
 boundary. Tracing a bad value backward through four engines in production 
 can take weeks. Catching the break early\, costs minutes. This session is 
 about closing that gap. As data becomes more relevant for analytics and AI
 \, that lack of visibility becomes a trust and compliance problem\, not on
 ly an operational one. This session demonstrates how to build a lineage gr
 aph across all four engines using OpenLineage and Marquez\, all open sourc
 e. We begin by demonstrating how to capture lineage from the streaming sou
 rce without modifying data records. We then fan a single Kafka topic out a
 cross parallel pipelines: an S3 sink connector for raw archival\, a real-t
 ime Flink DataStream job\, a Flink SQL job writing to Iceberg\, and a down
 stream Spark batch job that processes the Iceberg tables. We’ll show whe
 re the stack yields true column-level lineage versus dataset-level only\, 
 using OpenLineage's facet model. The talk is framed as a live investigatio
 n: a bad value surfaces in a Spark summary\, and we trace it backward thro
 ugh Iceberg\, Flink\, the Kafka topic\, and finally the source connector 
 — all visualized in Marquez. We compare this against Flink's native line
 age (FLIP-314)\, which stops at Flink's boundary\, to show why cross-engin
 e lineage matters. Attendees will leave with a runnable\, vendor-neutral b
 lueprint\, and an honest account of where open lineage tooling still falls
  short\, so they can apply these learnings in their production environment
 s. By attending this session\, your contact information may be shared with
  the sponsor for relevant follow up for this event only.
URL:https://sf26.rmoff.net/s/openlineage-to-track-them-all-from-kafka-to-th
 e-lakehouse-59
END:VEVENT
END:VCALENDAR