Skip to content

Case study · MILL5 · 2024 — 2026

An evaluation framework that catches LLM regressions in hours

Golden datasets, automated regression tests and quality metrics that turned “it feels worse” into a failing check.

Staff AI Engineer — design and implementation

Time to detect a model regression
9d → <4hTime to detect a model regression
Engineering workstreams on shared quality gates
4Engineering workstreams on shared quality gates

Context

Several teams were shipping features on top of the same LLM platform. Prompts, models and retrieval settings changed every week, but the only reliable signal that something had gotten worse was a user report — usually days after the change went out.

On average it took nine days for a regression to surface. By then it was hard to tell which of the dozen changes in that window had caused it.

Approach

I treated evaluation as part of the build, not a separate research activity. The goal was simple: any change to a prompt, model or retrieval setting should produce a score before it reaches users.

  • Built golden datasets per feature from real, reviewed interactions, with expected behaviour written down by the people who owned each feature.
  • Combined deterministic checks (format, citations, tool-call validity) with rubric-based model grading for open-ended answers, and spot-checked the graders against human labels.
  • Ran the suite automatically in CI on every relevant change, and on a schedule against production models to catch provider-side drift.
  • Tracked quality metrics over time so teams could see trends, not just pass/fail.

Results

  • Regression detection time dropped from 9 days to under 4 hours.
  • The same quality gates were adopted across 4 engineering workstreams, which made architecture reviews faster because everyone was reading the same numbers.
  • The framework later became the safety net for model routing work — it is what made it possible to move traffic to smaller models with confidence.

What I took from it

The hardest part was not the tooling, it was agreeing on what “good” meant for each feature. Writing that down early saved more time than any grader.

Stack

  • Python
  • Golden datasets
  • LLM-as-judge
  • CI/CD
  • GitHub Actions
  • Langfuse