Flink with an AI Agent That Debugs Itself: A Harness Engineering Walkthrough

Harness engineering is the production substrate that sits between an LLM agent and your live cluster: structured audit on every tool call, per-tool quotas, shadow mode, dry-run, human-in-the-loop approval for risky writes, trace propagation, retries, and a closed feedback loop from human PR review back into the agent's retrieval. It is what separates “we demoed the agent once” from “we left it on for six months and it kept the on-call asleep.” As streaming operations adopt autonomous agents, the harness — not the model — is the load-bearing engineering.

We built and run an AI debugging agent for Apache Flink. It ships as a JobManager sidecar in every Flink application-mode deployment: loopback to Flink REST, job-scoped Kubernetes permissions through a namespaced ServiceAccount, autoscaler state read from PostgreSQL, and inherited service identity for outbound calls. Per-deployment blast radius. Per-deployment audit trail. No central agent service, no fleet-wide credentials.

The diagnostic agent is a LangChain ReAct loop over eighteen Flink-specific tools: backpressure, checkpoint health, Kubernetes events, autoscaler state, thread dumps, and log search. It picks two to five tools based on the symptom. The harness wraps every tool call in eight composable LangChain middleware layers: audit, quota, shadow mode, dry-run, human-in-the-loop, model-call limit, retry, and trace propagation, using create_agent plus middleware composition. Tools are tagged by band — READ / WRITE / LEARN / OBSERVE — and risk tier — SAFE / MEDIUM / MANUAL — and the harness enforces tier-appropriate gates.

The agent is running in production at Yahoo today. When Grafana fires an alert, the Flink job debugs itself: the sidecar diagnoses the failure, the RCA lands in Slack, and the fix is still a PR a human reviews. Merged PRs train RAG positively; closed PRs flag patterns as “do not propose again.” The reviewer is the training signal — no model retraining required.

Attendees will leave with the eight-layer harness pattern, the JobManager-sidecar deployment model, the band-and-tier risk taxonomy, the closed-loop RAG design that turns PR review into agent training signal, and the failure modes to avoid, including how to protect the feedback loop from PR poisoning.