Intelligent LLM Routing: Quality-Gated Workload Dispatching

Traffic sent to a flagship model (GPT-4-class, Claude Sonnet/Opus-class) is not uniform in difficulty. Formatting a list, extracting three fields into JSON, and answering "what's the capital of Australia" are short, routine requests that a much cheaper model can serve. A default that sends everything to the most expensive model never tests whether that capability was required.

ACE's gateway sits in front of every model call, so it is the point at which a request can be inspected before dispatch. Classifying that request is the router's job.

How it decides

Every request first passes a fast, deterministic layer: explicit overrides, structural signals a cheap model routinely gets wrong (a code fence, an arithmetic or LaTeX expression), and configured keywords separating routine asks ("format," "extract," "summarize") from harder ones ("prove," "step by step," "debug"). Anything the router does not recognise defaults to the most capable model available — an error in the cheap direction is a quality regression, an error in the expensive direction costs what would have been paid anyway. The decision takes a couple of milliseconds: pattern matching only, no network call and no model inference on the hot path.

Above that sits a skill-aware router. It classifies what kind of work a request is — code, math, reasoning, extraction, translation, summarization, OCR, vision, long-context, ordinary chat — and how hard that instance is, then selects the cheapest model in the roster that clears a quality bar for that combination. Rather than a single cheap/flagship split, the selection covers every enabled model.

What is doing the classifying

Structural facts about a request — an image attached, a JSON response format requested, an unusually long input — are read first and treated as ground truth rather than predictions. Below that is a trained classifier: a compact model built on sentence embeddings, trained on hundreds of labeled examples across every skill category, reaching better than 97% held-out accuracy at separating routine requests from ones requiring reasoning. A second, independent scoring pass estimates difficulty and buckets each request into a low/medium/high tier, which maps onto the set of models worth calling.

The classifier is not itself a language model call, which would move the cost one level up rather than remove it. It is a small, fast, inspectable model that runs within the request's latency budget, and both the structural rules and a keyword fallback can override it. Rules and facts decide first; the learned model handles the ambiguous middle; the heuristic is the floor.

The escalation gate

Every cheap-tier answer is checked against a structural expectation — whether the JSON parsed, whether the output matches the requested shape. On failure the router retries once against the best available model and returns that result. The caller sees the correct answer, and the extra cost is incurred only on the retry path.

The cost of a wrong routing decision is therefore one additional call, not a degraded answer returned to the user. Each routing decision is recorded with the tier that served it, the reason, and whether it escalated.

Summary

The router's function is to stop paying for capability a request does not need without shipping a worse answer. Facts and structural checks decide first; a small trained classifier with a measured accuracy figure handles the ambiguous middle; the escalation gate bounds the downside. Further work extends the roster with more skills, more models, and finer-grained quality bars.

---

References

1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
2. I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
3. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618

---

Sign up to ACE now