Building an evals system for blue-collar AI quoting
Engineering deep-dive on Quotely, the photo-to-quote pipeline I commercialized from Moovez. How I split a vision + operations-research system into four eval layers, designed offline/online flywheels, and planned a harness that catches margin-killing quote errors before they reach dispatchers.
Founding PM · Moovez / Quotely
Eval harness design · ML Ops · Computer vision
Moving & last-mile logistics
2024 to Present
4 layers
Vision → catalog matching → logistics → end-to-end
Offline + online
CI regression + production quote accuracy flywheel
Deterministic first
Golden snapshots and invariants before Gemini spend
Honest status
Batch replay built; full harness package planned
Why this matters
A wrong quote in moving is not an abstract model failure. It means a crew shows up under-staffed, a truck doesn’t fit the load, or a dispatcher loses trust and overrides the system. Quotely turns customer photos into crew size, job duration, and price in under three minutes, but only if every layer in the pipeline is correct.
The obvious eval (compare final quote to a human dispatcher) hides where the miss happened. Was it vision (wrong item detected)? Catalog matching (right name, wrong weight class)? Or the calculator (stairs, elevator, crew logic)? I designed the eval architecture to answer that question layer by layer, not as one black box.
- Margin protection: 85% of jobs should finish within the quoted time window; systematic under-quoting erodes profitability
- Trust: Dispatchers override bad quotes; every override trains the org to distrust automation
- Commercialization: Quotely sells to other operators; quote accuracy is the product, not a nice-to-have metric
System under test
Quotely is a two-layer pipeline. Eval it in two layers first, then compose for end-to-end.
Layer 1 · Vision (non-deterministic)
Customer photos
↓ Gemini VLM
Item list: name, qty, size, location
- Statistical metrics: precision, recall, F1, quantity error
- Cross-run consistency (N=5 runs, report mean ± spread)
- Requires API spend; run on schedule or pre-release
Layers 2 and 3 · Logistics (deterministic)
Detected items
↓ Catalog enrichment (structured retrieval)
↓ MovingCalculator (OR engine)
Time, crew, price
- Golden-file snapshot tests: exact diff vs committed output
- Invariant assertions: crew bounds, truck sufficiency, monotonic pricing
- No Gemini required; runs free on every PR
Full pipeline:
Media → analyze_media() [Gemini]
→ enrich_items() [catalog lookup]
→ compute_logistics() [calculator]
→ quote
Four eval layers
Each layer answers a different question. Layers compose left to right; failures isolate to a single subsystem.
Layer 1 · Vision detection
Does the model see the right items?
Status: Planned automation. Consistency runs operational today.
- Ground truth: human-labeled inventories per photo set
- Metrics: recall, precision, F1, quantity MAE, size accuracy
- Matching: exact → alias → optional LLM-as-judge fallback
Layer 2 · Catalog matching
Do items map to the catalog correctly?
Status: Partially operational via batch test output.
- Deterministic, cheap, no Gemini
- Observable via calculationDebug.matching[]
- Signals I track: match rate, unknown-fallback leaderboard (which items need catalog additions), size resolution accuracy, and alias vs category method mix
- Triple-source compare (built): batch replay runs the same frozen vision output through three catalog backends (JSON, spreadsheet, and database) side by side to catch backend drift
Layer 3 · Logistics snapshots
Does the calculator produce the same output?
Status: Highest ROI; golden files planned.
- Frozen vision inputs replayed through enrichment + calculator
- Diff vs committed golden JSON (strip debug UUIDs)
- Invariants: crew bounds, truck sufficiency, phase sums
Layer 4 · End-to-end
Does photo → quote match reality?
Status: Partially operational via batch CSV actuals.
- Ground truth: customer-reported minutes, crew, price
- Metrics: minutes MAE, ±15% hit rate, crew mismatch, price-band hit
- 9 trusted historic jobs in headline set
Online vs offline evals
Two complementary halves: do not merge, do connect.
|
Offline (pre-ship) |
Online (production) |
| Question |
Is the system correct before we ship? |
Was the quote right in the real world? |
| Ground truth |
Labeled inventories, frozen outputs, historic job actuals |
Customer-reported minutes and crew after job completion |
| Speed |
Seconds to minutes (CI) |
Weeks (job completion cycle) |
| Coverage |
Every layer, isolated |
End-to-end totals only |
Flywheel: production drift surfaces bad quotes → new offline test cases → regression tests → fewer production failures. The calculationDebug object is shared: assert against it offline, persist it on quote records online.
Harness design
I specified a modular eval package: import production code, never reimplement calculator logic in test utilities.
- Design rules: Import enrich_items() and MovingCalculator from production. No Gemini for Layers 2 and 3. One report format: console table + JSON + Markdown.
- CI gates (planned): Every PR: logistics snapshot + matching + invariants hard-fail. Nightly: full vision (N=5) + E2E with alert if regression > ε vs 7-day baseline.
- Dataset A: Vision labels: Human-labeled inventories per photo set. Deprioritized until item-level debugging is needed for quote errors.
- Dataset C: Historic jobs: 14 frozen move fixtures joined to real job actuals. 9 trusted jobs in headline metrics; exclude guess, incomplete, and excluded quality tiers.
My role
As Founding PM on Quotely, I owned the product decision to split eval layers before scaling vision spend. I wrote the eval architecture spec, defined datasets and headline metrics aligned to dispatcher-trust and margin protection, and designed the harness package structure for the engineering team. The four-layer model came from diagnosing real quote failures, not from applying a generic ML evals template to a blue-collar domain.
Eval harness design
Ground-truth strategy
CI gate specification
Metric selection
Catalog gap analysis
Offline/online flywheel
Gemini + OR pipeline
Takeaways
- Split non-deterministic vision from deterministic logistics before you measure end-to-end quote accuracy
- Evaluate the retrieval layer you actually have: structured catalog matching, not hypothetical vector RAG
- Prioritize deterministic layers (golden snapshots, invariants) for highest ROI and free CI gates
- Measure what dispatchers feel as a “wrong quote” (minutes, crew, price band), not just model confidence
- Production drift is a dataset generator; every bad quote in the field should become an offline regression case
“In blue-collar AI, the eval question isn’t ‘is the model accurate?’ It’s ‘will the crew show up right-sized and the dispatcher trust the number?’”