BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-23@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261104T231500Z
DTEND:20261105T000000Z
SUMMARY:The Taming of Change Data Capture: Lessons from Slack's Streaming P
 ipelines
LOCATION:Room 2006
DESCRIPTION:Thomas Thornton\n\nSlack's previous batch data replication syst
 em cost millions of dollars per year and introduced up to 48 hours of late
 ncy. To solve both problems\, we replaced it with a real-time Change Data 
 Capture (CDC) pipeline built on Debezium\, Vitess\, Apache Kafka\, and Apa
 che Iceberg. That migration was just the beginning. What followed was four
  years of operating\, scaling\, and hardening these pipelines across thous
 ands of shards and petabytes of data in production. This talk is a lessons
 -learned walkthrough of what actually breaks when you run CDC and Kafka Co
 nnect at scale. We'll briefly cover the architecture\, then focus on the o
 perational challenges that apply to any streaming deployment: Reliability:
  How an unexpected interaction caused out-of-memory failures in production
 \, and the delivery pattern we adopted to prevent unbounded memory growth.
  Data loss prevention: A subtle edge case in Kafka Connect silently droppe
 d records for all connectors. How we detected & fixed it and what to monit
 or for your own pipelines. Load balancing: Why static connection assignmen
 t leaves throughput on the table\, and how periodic stream rotation with r
 esource-aware routing distributes work more evenly. Data safety: Automated
  enforcement of downstream size limits so one anomalous record does not ca
 scade into pipeline-wide failures. Monitoring and alerting: Which metrics 
 actually predict connector failures before they happen\, and which ones ar
 e noise. Autoscaling: Dynamically handling traffic spikes and petabyte-sca
 le table snapshots without over-provisioning or falling behind. Each lesso
 n came from a real production incident or bottleneck\, and led to contribu
 tions back to the open-source projects involved. More importantly\, the pa
 tterns are generalizable: they apply whether you're running Debezium\, ano
 ther CDC tool\, or any long-running streaming application. Attendees will 
 leave with a practical checklist for operating CDC\, Kafka Connect\, and d
 ata pipelines at scale. This includes monitoring strategies\, performance 
 tuning techniques\, resilience patterns\, and approaches to automating rec
 overy.
URL:https://sf26.rmoff.net/s/the-taming-of-change-data-capture-lessons-from
 -slacks-23
END:VEVENT
END:VCALENDAR