LLM Router v1: a scorecard

ACE's router decides, per request, which model to serve. v1 does this with three inputs and no call to a language model on the hot path:

1. a query-category classifier — what kind of work the request is,
2. a separate complexity scorer — how hard the instance is, and
3. the Model Market Matrix — the per-offering price and the capability score for each candidate model.

The first two are models we train and own — a multinomial logistic-regression category head and a ridge-regression complexity scorer, both running numpy-only inference over a shared 384-dimension bge-small sentence embedder, in-process, on CPU, with no network call and no external LLM in the loop. We control their weights, training data, and versioning, which is what lets us iterate on the numbers below rather than wait on a vendor.

The routing decision is: rank every model in the catalog by expected dollars, keep the ones whose capability at the request's category clears a complexity-dependent quality floor and that carry direct evidence for that category, and serve the cheapest one you can reach. If nothing qualifies, the request stays on the default model. A structural check on the response triggers a one-time retry against the strongest available model when the cheap answer fails.

Everything here is a v1 baseline — the first fully measured version of these heads, and the reference point we improve against. The figures come from the seed corpora and the router's own code paths; no production traffic is involved yet.

---

Scorecard 1 — the category classifier

The query-category head is a compact classifier over sentence embeddings. Held-out performance and a full-corpus regression pass (798 labeled prompts across 8 categories):

Per-category full-corpus accuracy:

The residual error is concentrated in one adjacency: 3 of the 6 misses are qa → chat, categories that route to similar tiers, so the dollar impact of the confusion is small. Structural facts (image attached, JSON response requested, input length) are read as ground truth and can override the learned head; a lexical rule is the floor under both.

Scorecard 2 — routing savings

Setup: the 798-prompt seed corpus (real benchmark prompts, 8 categories). For each prompt the router's own classifier and complexity scorer assign (category, complexity); the input leg is priced with the router's token estimator and the output leg with the catalog's per-category output priors.

• Baseline — every request served by one flagship, claude-opus-4-8 ($5 / $25 per 1M).
• Routed — cheapest catalog model with direct capability evidence that clears the complexity floor; otherwise the flagship is retained.

By category — the destinations, and the two categories the router declines to move:

The two 0.0% rows are the safety property in numbers. For qa no cheaper model carries a capability score, and for translation no model in the catalog carries one at all — so the router holds those on the flagship rather than route on a borrowed general-purpose score. We measured the alternative: allowing a general "chat" score to satisfy a hard floor collapses the policy to "always cheapest," which is the failure mode the evidence requirement removes.

Scorecard 3 — classification latency

The router adds one classification pass in front of every request, so its latency is a tax on the hot path. We timed the full in-house pass — embed with bge-small, category head, complexity head — over all 798 prompts, single-threaded on CPU (Apple Silicon dev host), steady-state after warmup:

The embedding forward pass dominates this budget; the two regression heads are numpy dot products and contribute microseconds. This is single-thread CPU with no batching, no GPU, and no ONNX thread tuning — the floor, not the ceiling. It sits well inside a normal request's latency budget, and because the heads are ours, batching and quantization are levers we hold for later versions.

Scorecard 4 — decision robustness

Two structural checks on whether the policy's decisions are stable, not just its dollar totals:

• Output-length invariance. Across all 55 model pairs in the current price set, one model is cheaper on both the input and output leg in 55 / 55 cases. No pair reverses ranking as the output grows, so the hand-estimated output priors move the reported dollar figure but not which model is selected. The decision is more stable than the exact savings number.
• Catalog vs. legacy routing table. The models the previous static routing matrix could rank and the models on the 2026-07 market table have an intersection of 0 (the single prefix match, gpt-5 ⊃ gpt-5.6-sol, is spurious). The router that predates the catalog was ranking a set of models disjoint from what the market actually offers — the measurement that motivated the catalog.

---

What this baseline establishes, and where v2 goes

These are the numbers for v1 — the first fully measured version of the two in-house heads. Because we own the training and serving, each one is a dial we can turn:

• The classifier's 0.975 held-out macro-F1 is a direct measurement. The residual error is one low-cost adjacency (qa → chat); a targeted data pass on that pair is the obvious next increment.
• The savings figure is a projection on benchmark prompts, not measured production savings. The corpus carries no completion tokens, so output length is a prior, and prompt lengths are short (p50 ≈ 38 tokens) relative to real traffic. Its magnitude also depends on the baseline — "everything to a $5/$25 flagship" is a high starting point; a deployment already on a mid-tier default sees a smaller headline. The category-level structure — which work moves, which is held — is the robust part, and calibrating the output priors from real traffic is what tightens the dollar figure.
• Latency is a floor, not a ceiling. Sub-10ms median on single-thread CPU leaves batching, quantization, and thread tuning as untouched levers.
• The cheap destinations' capability scores are still operator-supplied and unverified; promoting them to cited benchmark evidence is in progress and feeds directly back into which routes the evidence gate will allow.

Read the classifier accuracy and the latency as measurements; read the savings as a sourced projection with its assumptions stated. v2 iterates on all four.

---

References

1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
2. I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
3. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618