OpenLineage To Track Them All: From Kafka To The LakeHouse
Karel Alfonso Sague, Factor HouseChad Harris, Factor House
Wed, Nov 4, 2:00–2:45 PMRoom 2010
Most teams can draw their data architecture on a whiteboard but can't answer a basic governance question: when a value in a dataset looks wrong, where did it come from, and what downstream tables and models does it affect? In a stack spanning Kafka Connect, Flink, Spark, and Iceberg, lineage breaks at every engine boundary.
Tracing a bad value backward through four engines in production can take weeks. Catching the break early, costs minutes. This session is about closing that gap. As data becomes more relevant for analytics and AI, that lack of visibility becomes a trust and compliance problem, not only an operational one.
This session demonstrates how to build a lineage graph across all four engines using OpenLineage and Marquez, all open source. We begin by demonstrating how to capture lineage from the streaming source without modifying data records. We then fan a single Kafka topic out across parallel pipelines: an S3 sink connector for raw archival, a real-time Flink DataStream job, a Flink SQL job writing to Iceberg, and a downstream Spark batch job that processes the Iceberg tables. We’ll show where the stack yields true column-level lineage versus dataset-level only, using OpenLineage's facet model.
The talk is framed as a live investigation: a bad value surfaces in a Spark summary, and we trace it backward through Iceberg, Flink, the Kafka topic, and finally the source connector — all visualized in Marquez. We compare this against Flink's native lineage (FLIP-314), which stops at Flink's boundary, to show why cross-engine lineage matters.
Attendees will leave with a runnable, vendor-neutral blueprint, and an honest account of where open lineage tooling still falls short, so they can apply these learnings in their production environments.
By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.
Speakers
-
Karel Alfonso Sague
Staff Software Engineer, Factor House
Karel, Staff Engineer at Factor House in the Platform team, has more than 20 years experience in the IT industry. He excels at working across the data platform stack, particularly infrastructure applied to data engineering.
Prior to working at Factor House, Karel worked at Block for more than 8 years, across operational, streaming and analytical data platform teams. He brings expertise from multiple industries, having worked as a technical consultant for ThoughtWorks for more than 7 years.
He has presented at multiple conferences: Flink Forward 2023, Voxxed Days Australia, Programmable Australia 2023/2024 and local community events.
Outside of work Karel loves spending time with his family, hiking, bike riding and dancing.
-
Chad Harris
Solution Architect, Factor House
Chad Harris, Solutions Architect at Factor House, advises global customers on best practice to manage, govern and observe real-time data. His 25 years in architecture, software engineering, and engineering leadership are a rich resource for his clients.
Chad has run and built on Apache Kafka since its public release. He spent six years at Block leading Kafka and data teams across the organisation, his depth of knowledge extended to setting standards and direction for Iceberg. His background, including five years as Head of Engineering at a payments company, covers high-volume transactional systems, streaming and event driven architectures, tokenised security, and PCI compliance at scale.
He has presented at multiple conferences, including Voxxed Days Australia, Programmable Australia and local community events.
The war stories and actionable advice in this talk come from running Kafka in production for, and consulting with, some of the world's largest companies.