Lifting the Fog: Automated Kafka Topology Discovery for Event-Driven Architecture
Swati Bansal, Collectors
Thu, Nov 5, 1:30–1:45 PMInnovation Theater
Your Kafka cluster tells you everything about consumer behavior and nothing about producer identity. That asymmetry is by design — but at scale, it creates a fog that extends every incident, slows every migration, and makes orphaned infrastructure impossible to clean up safely. This session presents a graph-based Application Topology Agent that reconstructs the complete structural picture from two sources: broker metadata for the consumer side, and source code configuration for the producer side. Built in one week using AI-assisted development, the agent identified 42 orphaned consumer groups on first production run and reduced downstream impact triage from hours to 200 milliseconds. This is the architecture, the design decisions, and the lessons from running it in production.
Speaker
-
Swati Bansal
Principle Software Engineer, Collectors
With over 20 years of experience architecting mission-critical enterprise platforms and leading digital transformations, I work as a Principal Software Engineer at Collectors.
Throughout my career at organizations such as Collectors, MetLife, GrapeCity, and GlobalLogic (Hitachi Group Company), I have specialized in building high-scale digital ecosystems that power $Billion-GMV operations and my experience spans across:
- Architectural Excellence & Scale: Proven track record in platform engineering and microservices, including building an event driven intent-based platform that enabled over $500M in GMV and scaling the Vault business platform from 100k to 1.6M+ items.
- Cutting-Edge Innovation: Deep expertise in Agentic AI, RAG pipelines, and GenAI tool evaluation.
- Operational Impact: Led large-scale digital process automation initiatives generating $20M+ in workflow benefits and enterprise modernization efforts that achieved ~40% infrastructure cost reductions.
- Cross-Domain Versatility: Extensive experience across Marketplaces, Insurance, E-commerce, and Travel, providing me with a broad perspective to evaluate technology applications across diverse industries.
You can review my work experience at www.linkedin.com/in/swati-bansal-721b031b. You can check out some of my published work at : https://blog2.collectors.com/core-revolution-building-a-digital-engine-for-a-physical-world/