LLM Router v2: A Scorecard

In our earlier benchmark evaluation (LLM Router Out-of-Distribution Benchmark), we analyzed how static task classifiers behave under distribution shift, demonstrating that held-out in-distribution F1 scores (~0.975) dropped to 0.6058 (and 0.5199 under fallback heuristics) when tested against an independent, out-of-distribution (OOD) benchmark. Crucially, that shift exposed an under-service risk of over 51% on complex math and reasoning queries.

With LLM Router v2, we overhauled the classification engine, feature representation, and priority cascades. By introducing a multi-layer Feature Fusion MLP architecture—combining dense sentence embeddings, TF-IDF n-grams, and 12-dimensional rule vectors—alongside an expanded 7,665-prompt training corpus and refined lexical precedence, Router v2 lifts out-of-distribution macro-F1 to 0.7984 and overall accuracy to 77.0%.

When paired with ACE's dynamic Market Catalog policy (Arm A3), the system achieves 92.63% route agreement, $0.00 cost regret, and completely eliminates under-service risk to 0.00% across the entire 800-prompt OOD suite—all while operating locally on CPU with a median routing latency under 7 ms.

---

What Changed in Router v2 Architecture

The v2 routing pipeline operates on the hot path of every incoming API request, making deterministic decision choices before downstream tokens are generated. Inference remains 100% local on CPU with zero external network hops or recursive model calls.

Incoming Request
  │
  ├── 1. Structural Filter ──► Image / >32k Tokens / JSON ──► Direct Fast-Path Routing
  │
  ├── 2. Decisive Rule Precedence ──► High-Precision Regex (Math, Code, Reasoning)
  │
  ├── 3. Local Feature Fusion MLP  ──► Dense Embedding (bge-small 384-dim)
  │                                    ├── Sparse TF-IDF N-Grams
  │                                    ├── 12-Dim Rule Vector Features
  │                                    └── Multi-Layer Perceptron (GELU/ReLU)
  │
  └── 4. Market Catalog Policy ──► Capability Floor & Dynamic Cost Optimization

Key Architectural Upgrades

By fusing dense semantic representation (bge-small-en-v1.5) with sparse n-gram signals and explicit structural rule indicators, the classifier resolves fine-grained intent boundaries that semantic embeddings alone misinterpret (such as distinguishing formal proofs from general text Q&A).

---

Out-of-Distribution Benchmark Corpus

To rigorously validate transferability, we evaluate all router arms against a disjoint 800-prompt benchmark corpus. The dataset contains zero overlapping prompts or source templates from the router's training data.

                  ┌────────────────────────────────────────────────────────┐
                  │          800-Prompt Evaluation Benchmark               │
                  └───────────────────────────┬────────────────────────────┘
                                              │
         ┌────────────────────────────────────┼───────────────────────────────────┐
         │                                    │                                   │
┌────────▼─────────┐                 ┌────────▼─────────┐                ┌────────▼─────────┐
│ 520 Task Prompts │                 │ 120 Structural   │                │ 160 Adversarial  │
│ 8 Task Families  │                 │ Vision, OCR,     │                │ Stress Tests     │
│ Disjoint Sources │                 │ >32k Long-Ctx    │                │ JSON, Jargon, etc│
└──────────────────┘                 └──────────────────┘                └──────────────────┘

Dataset Breakdown

• 520 Core Task Prompts: Sourced from HumanEval, BigCodeBench, MATH, AQuA-RAT, SQuAD, MMLU-Pro, WMT16, and XSum across 8 gold categories: code (93), math (83), reasoning (82), qa (91), extraction (79), summarization (78), translation (83), and chat (91).
• 120 Structural Prompts: Verified multimodal image payloads, receipt/document OCR tasks, and long-context (>32k token) inputs.
• 160 Adversarial Prompts: 8 stress-test families $\times$ 20 prompts (including JSON-shaped non-extraction prompts, code-fence noise, multi-turn context drift, domain jargon, and polite conversational padding).

---

Empirical Benchmark Scorecard

We evaluated four execution configurations across the 800-prompt benchmark.

• Arm A0 (Keyword Heuristic): Regex & token-count rules (shipped fallback mode).
• Arm A1 (Centroid Fallback): Vector similarity against category centroids.
• Arm A2 (Trained Head): Local ONNX Feature Fusion MLP + Complexity Scorer.
• Arm A3 (Trained + Catalog): Feature Fusion MLP + Dynamic Market Catalog capability floors.

Key Performance Findings

1. Massive Accuracy Jump: Arm A2/A3 macro-F1 increased from 0.6058 to 0.7984 (+19.26 percentage points), with overall accuracy reaching 77.00% on complex out-of-distribution prompts.
2. Zero Under-Service Risk (Arm A3): Under-service rate—the share of requests routed to a model below the prompt's quality floor—dropped from 10.13% on legacy fallbacks down to exactly 0.00% on Arm A3.
3. Perfect Cost Regret: Arm A3 achieves $0.00 cost regret, meaning zero dollars are wasted on under-servicing tasks or selecting inefficient model tiers.

---

Deep Dive: Category Performance & Adversarial Robustness

Per-Category Performance Breakdown (Arm A2/A3)

The combination of lexical precedence and dense-sparse feature fusion significantly boosted previously difficult task domains:

Stress-Testing Adversarial Families

In v1, adversarial families like polite conversational padding or JSON-formatted code queries caused severe drift. Router v2 dramatically improves robustness across edge cases:

---

Latency Profile & Computational Efficiency

Router v2 preserves microsecond-level execution speeds by running native compiled ONNX matrix operations locally without thread contention or external network requests.

Evaluated on a standard single-thread server CPU instance (batch size = 1):

Relative to standard upstream LLM generation latency (~1,500 ms to 5,000 ms), the 6.85 ms local routing pass adds less than 0.22% overhead, ensuring zero perceptible delay for end-user requests.

---

Business Impact & Financial Economics

Precision routing directly impacts enterprise API economics. By automatically dispatching lightweight tasks (e.g., simple Q&A, structural summaries, translation) to high-throughput utility models while reserving flagship models (claude-3-5-sonnet, gpt-4o) for high-complexity math and reasoning, Router v2 delivers massive structural savings.

                  ┌────────────────────────────────────────────────────────┐
                  │           Cost Optimization Spectrum per 1M Reqs       │
                  └───────────────────────────┬────────────────────────────┘
                                              │
         ┌────────────────────────────────────┴───────────────────────────────────┐
         │                                                                        │
┌────────▼────────────────────────────────┐              ┌────────────────────────▼───────────────┐
│ Mid-Tier Default Baseline (gpt-4o)      │              │ Flagship Baseline (claude-3-5-sonnet)  │
│ Cost: $27,681 / 1M Requests             │              │ Cost: $180,139 / 1M Requests           │
└────────────────┬────────────────────────┘              └────────────────┬───────────────────────┘
                 │                                                        │
                 │  Routed Cost: $3,705 / 1M Reqs                          │  Routed Cost: $3,705 / 1M Reqs
                 ▼                                                        ▼
   86.62% Net Cost Reduction                                97.94% Net Cost Reduction
   ($23,976 Savings per 1M Reqs)                            ($176,434 Savings per 1M Reqs)

Financial Performance Breakdown

Crucially, because Arm A3 maintains a 0.00% under-service rate, these dollar savings reflect genuine structural efficiency rather than quality compromise.

---

The Road Ahead

LLM Router v2 demonstrates that combining lightweight local embeddings with feature-fusion classification and dynamic catalog policies allows API infrastructure to achieve 97.9% cost reductions while guaranteeing 100% quality floor compliance.

As model provider catalogs continue to diversify across specialized open-weights and proprietary APIs, local intent routing becomes a core primitive for enterprise AI platforms. We are actively expanding Router v2 with Item Response Theory (IRT) capability scoring and dynamic multi-turn session tracking.

To explore the benchmark suite and deployment guides, visit the ACE Fleet Documentation or inspect the open evaluation harnesses in our repository.

---

References

1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
2. I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
3. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618
4. I. Casanueva et al. Efficient Intent Detection with Pre-trained Sentence Encoders. ACL 2020. arXiv:2004.13845