BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-228@sf26-agenda
DTSTAMP:20261103T091709Z
DTSTART:20261104T212000Z
DTEND:20261104T215000Z
SUMMARY:How to NOT Evaluate an AI Agent
LOCATION:Innovation Theater
DESCRIPTION:Matt Searle\n\nAI agents are amazing - but give them a real Kaf
 ka cluster with all its real world problems and ask them to run a real job
 -to-be-done on a schedule with nobody watching. That's when things can get
  interesting. Here we will talk through our journey in eval testing agents
  for real clusters\, from tiny to silly-big; Agents running with their own
  credentials and their own budgets. We had some surprises on this journey\
 , and learned some unexpected things about our own MCP tools and about the
  models used. We also learned how an agent behaves when it thinks no one's
  checking\, and how hard it is to know whether a change actually made it b
 etter. This a story of evals for agents that live on top of Kafka. We'll c
 over the bits that broke and how we now catch them before our users do. By
  attending this session\, your contact information may be shared with the 
 sponsor for relevant follow up for this event only.
URL:https://sf26.rmoff.net/s/how-to-not-evaluate-an-ai-agent-228
END:VEVENT
END:VCALENDAR