BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-117@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T001500Z
DTEND:20261105T010000Z
SUMMARY:AI-Augmented Diagnostics for Change Data Capture Pipelines: Triage\
 , Lineage\, and Downstream Impact Assessment
LOCATION:Room 2010
DESCRIPTION:Jake Leister\n\nModern enterprise data platforms depend on Chan
 ge Data Capture (CDC) pipelines to deliver near-real-time data from operat
 ional source systems into analytical environments. When these pipelines fa
 il—due to producer pod crashes\, Kafka connector lag\, schema drift\, or
  Snowflake landing staleness—the impact cascades into delayed reporting\
 , broken dashboards\, and eroded stakeholder trust. We built a practical a
 rchitecture for AI-powered agents that autonomously support the triage\, d
 iagnosis\, and impact assessment of CDC pipeline failures across a multi-h
 op integration stack spanning SQL Server\, Kafka (Confluent)\, Kubernetes-
 hosted producers\, and Snowflake. We have leveraged this base in tools lik
 e cortex code\, claude and github copilot proving that spedning time in th
 e core design can be valuble no longer what tools you are using to impleme
 nt. The system leverages a Model Context Protocol (MCP) server and a curra
 ted knoweldge wiki that exposes read-only diagnostic tools to a large lang
 uage model orchestrator. These tools include SQL Server CDC status checks\
 , Kubernetes pod health and log retrieval\, Confluent topic and consumer-g
 roup lag inspection\, Snowflake sink connector status queries\, and observ
 ability integrations with Kibana\, Prometheus\, and Loki. A structured "st
 op-at-first-failure" health-check pattern walks each hop of the pipeline s
 equentially\, isolating the root cause without requiring the operator to m
 anually interrogate multiple systems. Critically\, the agent extends beyon
 d root-cause identification to downstream impact assessment. By querying t
 able and column lineage metadata\, dbt model dependency graphs\, and Power
  BI dataset refresh histories\, the agent identifies which marts\, reports
 \, dashboards\, applications and consuming teams are affected by a given p
 ipeline failure. This enables support engineers to proactively communicate
  impact scope\, prioritize remediation by business criticality\, and answe
 r the question stakeholders always ask first: "Who is affected and what do
  they see?" Governance gates enforce human approval before any state-chang
 ing operation—connector restarts\, job retries\, or deployments—ensuri
 ng the agent augments rather than replaces human judgment sending email me
 ssages or teams chats to alter it needs and intervention. The intended aud
 ience includes data engineers\, platform architects\, and engineering lead
 ers evaluating how AI assistants can reduce mean-time-to-resolution (MTTR)
  while preserving operational safety. Key takeaways include patterns for t
 ool design\, lineage-driven blast-radius analysis\, propose-before-execute
  governance\, and measurable improvements in triage speed when agents are 
 embedded in production support workflows.
URL:https://sf26.rmoff.net/s/ai-augmented-diagnostics-for-change-data-captu
 re-pipelines-117
END:VEVENT
END:VCALENDAR