LLM Router v1 Scorecard: Quantitative Benchmark of Model Tiering and Cost Savings

Out-of-Distribution Update (2026-08-08). The metrics in this post reflect in-distribution performance on a held-out split of the classifier's seed training corpus (0.975 macro-F1). To evaluate real-world generalization under distribution shift, we conducted an 800-prompt benchmark across 8 disjoint public datasets. For the latest Router v2 performance metrics, zero under-service guarantees, and feature fusion architecture, see LLM Router v2: A Scorecard.

ACE's router decides, per request, which model to serve. v1 uses three inputs and makes no call to a language model on the hot path:

1. a query-category classifier — what kind of work the request is,
2. a separate complexity scorer — how hard the instance is, and
3. the Model Market Matrix — the per-offering price and capability score for each candidate model.

The first two are trained in-house: a multinomial logistic-regression category head and a ridge-regression complexity scorer, both running numpy-only inference over a shared 384-dimension bge-small sentence embedder, in-process, on CPU, with no network call and no external LLM in the loop.

The routing decision: rank every model in the catalog by expected dollars, keep the ones whose capability at the request's category clears a complexity-dependent quality floor and that carry direct evidence for that category, and serve the cheapest one reachable. If nothing qualifies, the request stays on the default model. A structural check on the response triggers a one-time retry against the strongest available model when the cheap answer fails.

Everything here is a v1 baseline — the first fully measured version of these heads. The figures come from the seed corpora and the router's own code paths; no production traffic is involved.

---

Scorecard 1 — the category classifier

Held-out performance and a full-corpus regression pass (798 labeled prompts across 8 categories):

Per-category full-corpus accuracy:

The residual error is concentrated in one adjacency: 3 of the 6 misses are qa → chat, categories that route to similar tiers, so the dollar impact of the confusion is small. Structural facts (image attached, JSON response requested, input length) are read as ground truth and can override the learned head; a lexical rule is the floor under both.

Scorecard 2 — routing savings

Setup: the 798-prompt seed corpus (real benchmark prompts, 8 categories). For each prompt the router's own classifier and complexity scorer assign (category, complexity); the input leg is priced with the router's token estimator and the output leg with the catalog's per-category output priors.

• Baseline — every request served by one flagship, claude-opus-4-8 ($5 /$25 per 1M).
• Routed — cheapest catalog model with direct capability evidence that clears the complexity floor; otherwise the flagship is retained.

By category — the destinations, and the two categories the router does not move:

The two 0.0% rows are the evidence requirement in numbers. For qa no cheaper model carries a capability score, and for translation no model in the catalog carries one at all, so both are held on the flagship rather than routed on a borrowed general-purpose score. The alternative was measured: allowing a general "chat" score to satisfy a hard floor collapses the policy to "always cheapest".

Scorecard 3 — classification latency

The router adds one classification pass in front of every request. The full pass — embed with bge-small, category head, complexity head — timed over all 798 prompts, single-threaded on CPU, steady-state after warmup:

The embedding forward pass dominates the budget; the two regression heads are numpy dot products and contribute microseconds. This measurement is single-thread CPU with no batching, no GPU, and no ONNX thread tuning — a floor rather than a ceiling.

Scorecard 4 — decision robustness

Two structural checks on whether the policy's decisions are stable, independent of its dollar totals:

• Output-length invariance. Across all 55 model pairs in the current price set, one model is cheaper on both the input and output leg in 55 / 55 cases. No pair reverses ranking as the output grows, so the hand-estimated output priors move the reported dollar figure but not which model is selected.
• Catalog vs. the earlier routing table. The models the previous static routing matrix could rank and the models on the 2026-07 market table have an intersection of 0 (the single prefix match, gpt-5 ⊃ gpt-5.6-sol, is spurious). The router that predates the catalog was ranking a set of models disjoint from what the market offers.

---

What the baseline establishes

• The classifier's 0.975 held-out macro-F1 is a direct measurement. The residual error is one low-cost adjacency (qa → chat).
• The savings figure is a projection on benchmark prompts, not measured production savings. The corpus carries no completion tokens, so output length is a prior, and prompt lengths are short (p50 ≈ 38 tokens) relative to real traffic. The magnitude also depends on the baseline: "everything to a $5/$25 flagship" is a high starting point, and a deployment already on a mid-tier default would see a smaller headline. The category-level structure — which work moves, which is held — does not depend on the output priors.
• Latency is a floor, not a ceiling. Sub-10ms median on single-thread CPU is measured without batching, quantization, or thread tuning.
• The cheap destinations' capability scores are operator-supplied and unverified. Promoting them to cited benchmark evidence is in progress and changes which routes the evidence gate allows.

The classifier accuracy and the latency are measurements; the savings figure is a projection with its assumptions stated above.

---

References

1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
2. I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
3. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618

---

Sign up to ACE now