BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-8@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T223000Z
DTEND:20261105T231500Z
SUMMARY:When Kafka + Flink + Redis Isn't Enough: Building a Context Lake fo
 r Sub-Second Agent Decisions
LOCATION:Room 2012
DESCRIPTION:Xiaowei Jiang\n\nKafka and Flink are the foundation of every se
 rious real-time data system. Flink's stateful streaming\, combined with Ka
 fka's durability and Redis's low-latency reads\, gets teams most of the wa
 y to real-time decisioning. What breaks in production is the rest. The fai
 lure mode is context incoherence: two decisions evaluating the same entity
  at the same millisecond read from pipeline components at different propag
 ation stages. Your fraud model reads a Redis counter. Your authorization e
 ngine reads a Flink window aggregate. They're not synchronized\, and under
  burst load the gap is enough to approve transactions that should be block
 ed\, breach limits that should hold\, or trigger conflicting agent actions
  on shared state. Xiaowei Jiang\, creator of FlinkSQL\, shows this failure
  in production architectures at card issuers\, BNPL platforms\, and AI age
 nt orchestration systems running tens of thousands of decisions per minute
  — teams who built their pipelines on Flink and Kafka and still hit this
  wall. The root cause is structural: streaming pipelines deliver events in
 crementally\, but the derived state they produce propagates at different r
 ates across services. No amount of Flink tuning closes a gap that's archit
 ectural. Three patterns close it: Incrementally Maintained Views (IMVs) th
 at update sub-second and are read under a single coherent snapshot; a hybr
 id row+columnar layer that handles point lookups and analytical aggregatio
 ns without external caches; and a Postgres-compatible query interface that
  plugs into existing tooling without a stack rewrite. The talk covers what
  each pattern requires to implement\, where existing tools (Flink stateful
  functions\, Materialize\, Redis read-your-writes patterns) approximate it
  and where they fall short\, and the architectural boundary — the snapsh
 ot read layer — that Kafka + Flink + Redis is missing by design. Attende
 es leave with the name for the failure class (context incoherence)\, a dia
 gnostic for whether their pipeline is affected\, and a concrete architectu
 ral pattern to fix it.
URL:https://sf26.rmoff.net/s/when-kafka-flink-redis-isnt-enough-building-a-
 context-lake-8
END:VEVENT
END:VCALENDAR