Skip to content

Case study · MILL5 · 2024 — 2026

Model routing and semantic caching for LLM traffic

Sending each request to the smallest model that can handle it, and not paying twice for the same answer.

Staff AI Engineer — architecture and rollout

Reduction in inference cost
52%Reduction in inference cost
Of traffic moved to smaller models
70%Of traffic moved to smaller models
Daily AI requests served by the platform
10k+Daily AI requests served by the platform

Context

Every request on the platform went to the largest available model. It was the safe default while the product was young, but as usage grew past ten thousand requests a day, inference became one of the largest line items — and many requests were simple enough not to need it.

Approach

  • Classified incoming requests by task type and difficulty, and routed each class to the smallest model that met its quality bar on the evaluation suite.
  • Kept an automatic escalation path: low-confidence or failed responses were retried on a larger model instead of being returned to the user.
  • Added a semantic cache keyed on embeddings, with conservative similarity thresholds and per-feature rules about what was safe to cache.
  • Instrumented every hop with traces so cost, latency and quality could be broken down by route.

Results

  • 70% of AI traffic moved to smaller models without measurable quality degradation on the evaluation suite.
  • Inference cost fell by 52%.
  • Average latency improved as a side effect, since smaller models and cache hits respond faster.

What I took from it

Routing is only as trustworthy as the evaluation behind it. Without the regression suite, this would have been a guess; with it, each routing rule was a measured decision that could be rolled back.

Stack

  • Python
  • FastAPI
  • Redis
  • Embeddings
  • OpenAI
  • Anthropic Claude
  • OpenTelemetry