BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-48@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T175000Z
DTEND:20261105T180500Z
SUMMARY:Breaking the Bottleneck: Why We Built Our Own Kafka Connect Source 
 for Massive Mainframe Offloading
LOCATION:Innovation Theater
DESCRIPTION:Marco Antonio Jonack\n\nMainframes aren't going away — but th
 eir data needs to be everywhere\, in real time. For decades\, the mainfram
 e has been the undisputed workhorse of Brazil’s national IT infrastructu
 re\, holding the most critical data for over 200 million citizens. At Serp
 ro\, Brazil's largest state-owned IT company (founded over 60 years ago an
 d responsible for many mission-critical government systems)\, we embarked 
 on a massive Mainframe Offload initiative to modernize our architecture\, 
 reduce legacy costs\, and replicate mission-critical data to distributed t
 echnologies. Like many large enterprises\, our initial approaches fell sho
 rt. Homegrown batch solutions completely failed to meet real-time requirem
 ents. We then turned to a leading proprietary CDC (Change Data Capture) en
 terprise tool\, only to hit severe operational limits at national scale: i
 nitial loads took days for specific datasets\, the architecture lacked hor
 izontal scalability (no partitioned data processing)\, and per-destination
  licensing costs were prohibitive. We decided to build our own. In this se
 ssion\, we will deep dive into how we engineered a custom Kafka Connect So
 urce Connector tailored specifically for massive-scale mainframe replicati
 on. Kafka and Kafka Connect now serve as the distributed backbone for inge
 sting\, retaining\, and distributing mainframe data to any downstream tech
 nology — without vendor lock-in. Built from the ground up with paralleli
 sm\, partitioning\, and extreme throughput in mind\, our custom solution c
 ompletely outperformed the proprietary tool. Head-to-head benchmarks showe
 d: Over 500% improvement in real-time replication throughput. Nearly 250% 
 speedup for initial data loads (bringing loads from days to hours). Audien
 ce Takeaways: Architecting for Scale: Design and optimize a custom Kafka C
 onnect Source Connector for high-throughput Mainframe CDC. Overcoming Bott
 lenecks: Key design patterns for parallelizing and partitioning legacy dat
 a streams to solve the pain points of initial loads vs. real-time synchron
 ization. Real-World Impact: Practical lessons learned when replacing a com
 mercial replication product in a government production environment\, trans
 forming silos into event-driven data products.
URL:https://sf26.rmoff.net/s/breaking-the-bottleneck-why-we-built-our-own-k
 afka-connect-48
END:VEVENT
END:VCALENDAR