Case study · MILL5 · 2024 — 2026
Model routing and semantic caching for LLM traffic
Sending each request to the smallest model that can handle it, and not paying twice for the same answer.
Staff AI Engineer — architecture and rollout
- Reduction in inference cost
- 52%Reduction in inference cost
- Of traffic moved to smaller models
- 70%Of traffic moved to smaller models
- Daily AI requests served by the platform
- 10k+Daily AI requests served by the platform
Context
Every request on the platform went to the largest available model. It was the safe default while the product was young, but as usage grew past ten thousand requests a day, inference became one of the largest line items — and many requests were simple enough not to need it.
Approach
- Classified incoming requests by task type and difficulty, and routed each class to the smallest model that met its quality bar on the evaluation suite.
- Kept an automatic escalation path: low-confidence or failed responses were retried on a larger model instead of being returned to the user.
- Added a semantic cache keyed on embeddings, with conservative similarity thresholds and per-feature rules about what was safe to cache.
- Instrumented every hop with traces so cost, latency and quality could be broken down by route.
Results
- 70% of AI traffic moved to smaller models without measurable quality degradation on the evaluation suite.
- Inference cost fell by 52%.
- Average latency improved as a side effect, since smaller models and cache hits respond faster.
What I took from it
Routing is only as trustworthy as the evaluation behind it. Without the regression suite, this would have been a guess; with it, each routing rule was a measured decision that could be rolled back.
Stack
- Python
- FastAPI
- Redis
- Embeddings
- OpenAI
- Anthropic Claude
- OpenTelemetry