One Hash, N Connectors: Parallelizing MongoDB Managed Source Connectors Without Writing Connector Code
Guilherme Ferreira, Murmuration
Thu, Nov 5, 11:30–11:45 AMInnovation Theater
Confluent Cloud's fully managed MongoDB source connector is easy to operate; however, it is limited to one task per connector. When we needed to move nearly a billion documents (around 20TB of data) from MongoDB to Apache Pinot through Kafka each month, that limit became a serious bottleneck.
In this lightning talk, I'll show how we turned one connector into N. Using hash-based filtering in MongoDB's aggregation pipeline, we assigned multiple managed connectors predictable, non-overlapping slices of the same collection. No custom connector code, plugins, or coordination service required.
You'll learn where this pattern fits, how we applied it to both historical backfills and change data capture, and the production gotchas we wish we had known about from the start. If a single managed connector is limiting your migration or streaming throughput, you'll leave with a practical pattern for scaling horizontally while keeping the operational benefits of fully managed infrastructure.
Speaker
-
Guilherme Ferreira
Principal Data Engineer, Murmuration
Guilherme Ferreira is a Principal Data Engineer at Murmuration, a nonprofit civic tech organization that provides data and technology to strengthen community organizing and democratic participation in the United States. Over the past year, he designed and built the streaming infrastructure that migrates and syncs close to a billion documents from MongoDB into Apache Pinot through Confluent Cloud, now running in production. With 10 years of experience in data engineering, his work focuses on building scalable data platforms that serve real-world impact. Guilherme has attended Current for the past two years and is excited to present for the first time.