Falcon: Democratizing Stream Processing with Flink SQL and Config-Driven Event Routing

Core Theme & Relevance At Slack, we process 100s billions of events daily, but accessing that data historically in realtime felt like searching through an unthreaded channel—requiring deep Java expertise, complex wire formats, and weeks of waiting. This talk introduces Falcon, our internal Flink SQL platform that democratizes stream processing. We'll share how we empowered any engineer to go from a data request to a running streaming query in minutes, making real-time data as accessible as sending a Slack message.

Technical Depth We will dive into Falcon's architecture, highlighting our config-driven event router and SQL-first development experience. We'll explore how our event router uses YAML-defined rules to seamlessly untangle high-fanout Kafka topics into dedicated streams, utilizing sample rates for safe rollouts and cost saving. We’ll also demonstrate our dynamic Thrift path resolver for Flink SQL, which brings submission-time schema validation to nested struct fields, catching DDL typos early instead of failing silently at runtime.

Real-World Connection & Audience Takeaways Attendees will leave with actionable patterns for building self-service streaming platforms that developers actually enjoy using. Key takeaways include Slack's strategies for balancing SQL ergonomics with custom format connectors, techniques for operating config-driven routers at massive scale, and architectural principles for designing control planes that abstract Kubernetes complexity while maintaining deep observability.