A technical showcase on choosing eval metrics for agents with retrieval and human escalation. When Cohen’s κ belongs in the stack, how to measure HITL quality, and why V1 ship gates should be task-success floors on fixtures, not mislabeled confidence scores. Illustrated with Moovez’s SMS support agent.
Cohen’s κ measures chance-corrected agreement between two independent raters labeling the same items (two humans, or a human and an LLM judge on the same turns).
It requires paired labels, a confusion matrix, and enough cases to be stable. The right use in agent evals is often offline calibration: “Does our LLM judge agree with humans beyond chance?” That is a reliability study, not a product ship gate.
On Moovez, the prior bot’s “kappa” was really a confidence threshold. We kept the useful idea as Stage 2 confidence plus policy rules and explicit-human requests, and reserved true κ for a future judge-calibration pass. Standard deviation across runs can describe flaky live metrics later; we do not use SD as a primary V1 gate.
Human-in-the-loop is not one failure mode. Missed escalations and noisy escalations hurt different parts of the business and need separate metrics.
Escalation triggers should be explicit in design and in fixtures:
A single “escalation rate” hides whether you are failing open (missed) or failing closed (noisy). Track recall and precision on labeled fixtures, not one blended number.
Moovez’s SMS agent handles intake, booking confirmations, reschedules, and routine support over text, with retrieval-backed replies and escalation to humans when needed. The product goal was meaningful support deflection (~50% ticket deflection in the broader Moovez case) without inventing booking state or hiding cases that needed a dispatcher.
V1 ship gates on a fixed Moovez fixture suite (pass-rate floors, not agreement statistics):
intake_complete without hard escalated when the fixture expects a soft pathThis pattern transfers to any retrieval + escalation agent: define task success per fixture, separate HITL failure modes, and use κ only when you have paired labels for judge calibration.
As Founding PM, I specified which metrics belong in V1 ship gates vs later calibration work, defined the Moovez fixture suite (hit@4, HITL P/R, intake / soft-handoff), and aligned the team on Stage 2 confidence + policy escalation instead of mislabeled “kappa.” The framework is product-agnostic; the SMS agent is where we proved it.
“For conversational agents, the product question is whether the task finished safely and the right cases reached a human. Agreement statistics answer a different question, and only when you have paired labels.”