When Kafka + Flink + Redis Isn't Enough: Building a Context Lake for Sub-Second Agent Decisions

Kafka and Flink are the foundation of every serious real-time data system. Flink's stateful streaming, combined with Kafka's durability and Redis's low-latency reads, gets teams most of the way to real-time decisioning. What breaks in production is the rest.

The failure mode is context incoherence: two decisions evaluating the same entity at the same millisecond read from pipeline components at different propagation stages. Your fraud model reads a Redis counter. Your authorization engine reads a Flink window aggregate. They're not synchronized, and under burst load the gap is enough to approve transactions that should be blocked, breach limits that should hold, or trigger conflicting agent actions on shared state.

Xiaowei Jiang, creator of FlinkSQL, shows this failure in production architectures at card issuers, BNPL platforms, and AI agent orchestration systems running tens of thousands of decisions per minute — teams who built their pipelines on Flink and Kafka and still hit this wall. The root cause is structural: streaming pipelines deliver events incrementally, but the derived state they produce propagates at different rates across services. No amount of Flink tuning closes a gap that's architectural.

Three patterns close it: Incrementally Maintained Views (IMVs) that update sub-second and are read under a single coherent snapshot; a hybrid row+columnar layer that handles point lookups and analytical aggregations without external caches; and a Postgres-compatible query interface that plugs into existing tooling without a stack rewrite. The talk covers what each pattern requires to implement, where existing tools (Flink stateful functions, Materialize, Redis read-your-writes patterns) approximate it and where they fall short, and the architectural boundary — the snapshot read layer — that Kafka + Flink + Redis is missing by design.

Attendees leave with the name for the failure class (context incoherence), a diagnostic for whether their pipeline is affected, and a concrete architectural pattern to fix it.