The Taming of Change Data Capture: Lessons from Slack's Streaming Pipelines

Slack's previous batch data replication system cost millions of dollars per year and introduced up to 48 hours of latency. To solve both problems, we replaced it with a real-time Change Data Capture (CDC) pipeline built on Debezium, Vitess, Apache Kafka, and Apache Iceberg. That migration was just the beginning. What followed was four years of operating, scaling, and hardening these pipelines across thousands of shards and petabytes of data in production.

This talk is a lessons-learned walkthrough of what actually breaks when you run CDC and Kafka Connect at scale. We'll briefly cover the architecture, then focus on the operational challenges that apply to any streaming deployment:

• Reliability: How an unexpected interaction caused out-of-memory failures in production, and the delivery pattern we adopted to prevent unbounded memory growth.
• Data loss prevention: A subtle edge case in Kafka Connect silently dropped records for all connectors. How we detected & fixed it and what to monitor for your own pipelines.
• Load balancing: Why static connection assignment leaves throughput on the table, and how periodic stream rotation with resource-aware routing distributes work more evenly.
• Data safety: Automated enforcement of downstream size limits so one anomalous record does not cascade into pipeline-wide failures.
• Monitoring and alerting: Which metrics actually predict connector failures before they happen, and which ones are noise.
• Autoscaling: Dynamically handling traffic spikes and petabyte-scale table snapshots without over-provisioning or falling behind.

Each lesson came from a real production incident or bottleneck, and led to contributions back to the open-source projects involved. More importantly, the patterns are generalizable: they apply whether you're running Debezium, another CDC tool, or any long-running streaming application.

Attendees will leave with a practical checklist for operating CDC, Kafka Connect, and data pipelines at scale. This includes monitoring strategies, performance tuning techniques, resilience patterns, and approaches to automating recovery.