How to NOT Evaluate an AI Agent

AI agents are amazing - but give them a real Kafka cluster with all its real world problems and ask them to run a real job-to-be-done on a schedule with nobody watching. That's when things can get interesting. Here we will talk through our journey in eval testing agents for real clusters, from tiny to silly-big; Agents running with their own credentials and their own budgets. We had some surprises on this journey, and learned some unexpected things about our own MCP tools and about the models used. We also learned how an agent behaves when it thinks no one's checking, and how hard it is to know whether a change actually made it better. This a story of evals for agents that live on top of Kafka. We'll cover the bits that broke and how we now catch them before our users do.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.