These are not the brokers you are connecting to

Every platform team has dreamt of splitting an Apache Kafka cluster, but the reality is far messier. Application teams think in topics and clusters, but Kafka forces them to care about broker topology. We can't fix that, but this talk is about giving platform teams the tools required to make that dream come true.

Most proxies operate on Kafka at Layer 4 — they forward connections, not messages. Kroxylicious works at Layer 7: it understands individual Kafka protocol messages and controls the broker topology the client sees. Multiple physical clusters can appear as one, or one cluster can appear as many — the client's view is whatever the platform team decides it should be.

The foundation for this is Kroxylicious's new routing API — a plugin interface that lets you build custom routing logic as first-class extensions. Router authors decide where messages go; the proxy handles making the broker topology look coherent to the client.

This enables a family of patterns: Union Clusters — topics from multiple physical clusters presented as one; Union Topics — a single topic sharded across clusters; and Cluster Aliasing — presenting one cluster under the identity of another. It also lets the proxy transparently leverage KIP-392 for AZ-aware fetching.

This talk will show you how the routing API works, walk through the patterns routers enable, and give you everything you need to build a router. All you need to bring is your own use case.