Breaking the Bottleneck: Why We Built Our Own Kafka Connect Source for Massive Mainframe Offloading

Mainframes aren't going away — but their data needs to be everywhere, in real time. For decades, the mainframe has been the undisputed workhorse of Brazil’s national IT infrastructure, holding the most critical data for over 200 million citizens.

At Serpro, Brazil's largest state-owned IT company (founded over 60 years ago and responsible for many mission-critical government systems), we embarked on a massive Mainframe Offload initiative to modernize our architecture, reduce legacy costs, and replicate mission-critical data to distributed technologies.

Like many large enterprises, our initial approaches fell short. Homegrown batch solutions completely failed to meet real-time requirements. We then turned to a leading proprietary CDC (Change Data Capture) enterprise tool, only to hit severe operational limits at national scale: initial loads took days for specific datasets, the architecture lacked horizontal scalability (no partitioned data processing), and per-destination licensing costs were prohibitive.

We decided to build our own. In this session, we will deep dive into how we engineered a custom Kafka Connect Source Connector tailored specifically for massive-scale mainframe replication. Kafka and Kafka Connect now serve as the distributed backbone for ingesting, retaining, and distributing mainframe data to any downstream technology — without vendor lock-in.

Built from the ground up with parallelism, partitioning, and extreme throughput in mind, our custom solution completely outperformed the proprietary tool. Head-to-head benchmarks showed:

• Over 500% improvement in real-time replication throughput.
• Nearly 250% speedup for initial data loads (bringing loads from days to hours).

Audience Takeaways:

• Architecting for Scale: Design and optimize a custom Kafka Connect Source Connector for high-throughput Mainframe CDC.
• Overcoming Bottlenecks: Key design patterns for parallelizing and partitioning legacy data streams to solve the pain points of initial loads vs. real-time synchronization.
• Real-World Impact: Practical lessons learned when replacing a commercial replication product in a government production environment, transforming silos into event-driven data products.