Don't pay flagship prices for a regex
Most requests hitting your most expensive model don't need it. ACE's router classifies each request's actual difficulty and sends routine work to a cheap model — with a quality gate that catches it when the cheap model isn't good enough, so cheapening the default never means degrading.
Don't pay flagship prices for a regex
Here's a number worth sitting with: a large share of the traffic hitting a flagship model (GPT-4-class, Claude Sonnet/Opus-class) is short, routine, and would come back exactly as correct from a model that costs a fraction as much. Formatting a list. Extracting three fields into JSON. Answering "what's the capital of Australia." Sending that to your most expensive model isn't a quality choice — it's just not checking whether you needed to.
Paying flagship pricing for routine work is like paying a senior engineer's hourly rate to file paperwork. The work still gets done. You just didn't need to spend that much to get it done.
ACE's gateway sits in front of every model call, which means it's the one place that can look at a request before it goes anywhere and ask: does this actually need the expensive model? That's the whole job of the router.
How it decides
Every request first passes through a fast, deterministic layer of checks — explicit overrides, structural signals a cheap model routinely gets wrong (a code fence, an arithmetic or LaTeX expression), and configured keywords that separate routine asks ("format," "extract," "summarize") from harder ones ("prove," "step by step," "debug"). Anything the router doesn't recognize defaults to the most capable model available — the router never silently downgrades work it can't confidently classify, because guessing wrong in the cheap direction is a quality regression nobody signed up for, while guessing wrong in the expensive direction just costs what you'd have paid anyway. This whole decision happens in a couple of milliseconds — pure pattern matching, no network call, no model inference on the hot path.
Layered on top of that is a richer, skill-aware router: it classifies what kind of work a request actually is — code, math, reasoning, extraction, translation, summarization, OCR, vision, long-context, ordinary chat — and how hard an instance of that skill is, then picks the cheapest model in your roster that clears a quality bar for that specific combination. Instead of a single cheap/flagship split, it's the actual cheapest adequate option across everything you've enabled.
A real classifier, not a keyword list dressed up as one
"AI-powered routing" is an easy phrase to oversell, so it's worth being precise about what's actually deciding here. Structural facts about a request — an image attached, a JSON response format requested, an unusually long input — are read first and treated as ground truth, not predictions. Underneath that sits a genuinely trained classifier: a compact model built on sentence embeddings, trained on hundreds of labeled examples across every skill category, that reaches better than 97% held-out accuracy at telling routine requests from ones that actually need reasoning power. A second, independent scoring pass estimates difficulty the same way, bucketing each request into a low/medium/high tier that maps onto which models are worth calling at all.
This is deliberately not "have a language model classify your prompt" — that would just move the cost you're trying to save one level up. It's a small, fast, inspectable classifier that runs comfortably within the request's overall latency budget, with hard structural rules and an even-cheaper keyword fallback both able to override it. Rules and facts decide first because they're certain; the learned model handles the ambiguous middle; a plain heuristic is the floor under everything.
The part that makes cheapening safe: the escalation gate
None of the above matters if routing to a cheap model sometimes just returns a worse answer with no recourse. The router's real safety property is what happens after the cheap model responds: every cheap-tier answer is checked against a structural expectation — did the JSON actually parse, does the output look like what was asked for — and if it fails, the router retries exactly once against the best available model and returns that instead. The caller never sees the failure; they get a correct answer, having only paid extra on the retry path when the cheap attempt genuinely wasn't good enough.
This is what makes "route to cheap by default" a decision worth trusting rather than a gamble: the downside of guessing wrong is one extra call, never a silently worse answer shipped to the user. Every routing decision is also fully explainable after the fact — which tier served it, why, and whether it escalated — so "why did this land on that model" always has an answer.
The takeaway
The router's job isn't to be clever. It's to stop paying for capability a request doesn't need, without ever silently shipping a worse answer to get there. Facts and structural checks decide first because they're certain, not guesses; a small trained classifier handles the ambiguous middle with a number you can audit; and the one thing that's never optional is the escalation gate that catches the cheap model when it's wrong. Cheap-by-default only works if wrong-by-default is impossible — and that's the property the rest of ACE's routing roadmap builds outward from, as more skills, more models, and finer-grained quality bars get added to the roster it chooses from.
References
- L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
- I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
- D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618