Least Privilege for Non-Human Identities: Governing an Autonomous Kafka SRE Agent

Anyone can build an agent that suggests a fix for a Kafka incident. The hard part is letting it act on production, where one bad call can purge a topic or reset consumer offsets and take real data down with it.

This is a tales-from-the-trenches talk about an SRE agent we built to operate Kafka, and mostly about the part nobody puts in the demo: how we permissioned it so that even a fully compromised agent can't do catastrophic damage.

A Kafka SRE agent is close to a worst case. It reads private telemetry. It ingests content we don't control, so a single poisoned log line becomes a prompt-injection vector. And it can do things you can't take back. So we stopped trusting the model to be careful and moved the authority into the permission layer instead. The model only ever gets to recommend.

The model we landed on, after a couple of incidents taught us better, tiers actions by blast radius. Read-only diagnostics like consumer lag, under-replicated partitions, and broker health run on their own. Reversible actions like restarting a connector task or nudging a quota run too, but everything gets logged. The destructive stuff (deleting topics, resetting offsets, reassigning partitions) always waits for a human. Under all of it, the agent has its own least-privilege identity, scoped per operation with Kafka ACLs instead of borrowing an admin's credentials. Approval gates show the predicted blast radius before you say yes, we dry-run what we can, and an append-only log ties every action back to a person.

We'll also get into what went wrong: an action the agent proposed off a signal that turned out to be misleading, a permission we scoped too wide and had to claw back, and how fast the non-human identities piled up once we stopped counting. The thing we kept relearning the hard way: being allowed to connect to a system isn't the same as being allowed to act on it.

You'll leave with the actual permission model, tiered by blast radius, and it works for any agent touching production, not just a Kafka one.