Evaluating conversational agents: HITL and agreement metrics

A technical showcase on choosing eval metrics for agents with retrieval and human escalation. When Cohen’s κ belongs in the stack, how to measure HITL quality, and why V1 ship gates should be task-success floors on fixtures, not mislabeled confidence scores. Illustrated with Moovez’s SMS support agent.

Founding PM · Moovez Conversational agents · Eval design · HITL Agreement metrics · Fixture gates 2024 to Present
When κ applies
Paired raters, confusion matrix, chance correction
HITL quality
Missed vs noisy escalations as separate failures
Task-success floors
Fixture pass rates as V1 ship gates
Worked example
Moovez SMS agent (intake, booking, deflection)

When Cohen’s kappa applies (and when it doesn’t)

Cohen’s κ measures chance-corrected agreement between two independent raters labeling the same items (two humans, or a human and an LLM judge on the same turns).

It requires paired labels, a confusion matrix, and enough cases to be stable. The right use in agent evals is often offline calibration: “Does our LLM judge agree with humans beyond chance?” That is a reliability study, not a product ship gate.

True κ (use later)

  • Two raters, same turns, categorical labels
  • Confusion matrix + chance correction
  • Judge vs human calibration on a gold set
  • Dual-rater inter-rater reliability when you trust the judge enough to automate

Not κ (common mistake)

  • LLM confidence score ∈ [0, 1] used as a threshold
  • Single-model score with no paired second rater
  • Calling that threshold “kappa” in dashboards or specs
  • Using agreement stats when the gate is “did the task succeed?”

On Moovez, the prior bot’s “kappa” was really a confidence threshold. We kept the useful idea as Stage 2 confidence plus policy rules and explicit-human requests, and reserved true κ for a future judge-calibration pass. Standard deviation across runs can describe flaky live metrics later; we do not use SD as a primary V1 gate.

Designing HITL quality metrics

Human-in-the-loop is not one failure mode. Missed escalations and noisy escalations hurt different parts of the business and need separate metrics.

Missed escalation (recall)

  • Case should have reached a human; bot kept going
  • Customer stuck with wrong booking status or bad reply
  • Burns trust; hard to recover in SMS
  • Metric: recall on “should escalate” fixtures

Noisy escalation (precision)

  • Human pulled in when the bot could have finished
  • Ops capacity wasted on routine intake or reschedules
  • Deflection target missed; support cost scales with volume
  • Metric: precision on escalation decisions

Escalation triggers should be explicit in design and in fixtures:

A single “escalation rate” hides whether you are failing open (missed) or failing closed (noisy). Track recall and precision on labeled fixtures, not one blended number.

Worked example: Moovez SMS agent

Moovez’s SMS agent handles intake, booking confirmations, reschedules, and routine support over text, with retrieval-backed replies and escalation to humans when needed. The product goal was meaningful support deflection (~50% ticket deflection in the broader Moovez case) without inventing booking state or hiding cases that needed a dispatcher.

Inbound SMS ↓ Retrieval (top-k chunks) ↓ Reply / intake step / soft handoff ↓ Stage 2 confidence + policy + explicit-human check ↓ HITL queue or closed loop

V1 ship gates on a fixed Moovez fixture suite (pass-rate floors, not agreement statistics):

This pattern transfers to any retrieval + escalation agent: define task success per fixture, separate HITL failure modes, and use κ only when you have paired labels for judge calibration.

My role

As Founding PM, I specified which metrics belong in V1 ship gates vs later calibration work, defined the Moovez fixture suite (hit@4, HITL P/R, intake / soft-handoff), and aligned the team on Stage 2 confidence + policy escalation instead of mislabeled “kappa.” The framework is product-agnostic; the SMS agent is where we proved it.

Agreement metrics (κ) HITL recall / precision Fixture design Confidence thresholds LLM-as-judge calibration Conversational agent evals

Takeaways

“For conversational agents, the product question is whether the task finished safely and the right cases reached a human. Agreement statistics answer a different question, and only when you have paired labels.”