LLM Router v2: A Scorecard with Automated Model Market Catalog Integration
An updated 800-prompt out-of-distribution evaluation of ACE's local ONNX LLM router: analyzing feature fusion classification (0.798 macro-F1), 300+ model pricing catalog ingestion, 0.00% under-service risk, and 86.6%-97.9% cost savings.
In our earlier benchmark evaluations, we analyzed how static task classifiers behave under distribution shift, demonstrating that held-out in-distribution F1 scores (~0.975) dropped significantly when tested against an independent, out-of-distribution (OOD) benchmark. Crucially, legacy keyword routing exposed an under-service risk of over 10.1% overall and > 53% on complex math and reasoning queries.
With LLM Router v2, we overhauled the classification engine, feature representation, and market routing policies. By integrating the Automated Model Market Catalog alongside a multi-layer Feature Fusion MLP architecture—combining dense sentence embeddings, TF-IDF n-grams, and 12-dimensional rule vectors—Router v2 achieves 0.7984 macro-F1, 77.0% classification accuracy, and 92.63% route agreement.
When running under Arm A3 (Trained Classifier + Market Matrix), the system achieves $0.00 cost regret, 86.62% dollar savings against a mid-tier gpt-4o baseline, and completely eliminates under-service risk to 0.0000% (0 under-served requests out of 800)—all operating locally on CPU with a median routing latency under 7 ms.
1. 800-Prompt Benchmark Evaluation Summary
We evaluated four router execution arms against an independent, 800-prompt disjoint evaluation corpus (data/router_eval/router_eval.jsonl):
+-------------------------------------------------------------------------------------------------------------------------+
| LLM ROUTER V2 SCORECARD (800-PROMPT OOD BENCHMARK) |
+-------------------------------+-----------------------+---------------------+-------------------+-----------------------+
| Router Execution Arm | Macro-F1 Score | Classification Acc | Route Agreement | Under-Service Rate |
+-------------------------------+-----------------------+---------------------+-------------------+-----------------------+
| A0: Keyword (Legacy Deployed) | 0.6552 | 60.38% | 87.25% | 10.13% (81 / 800) |
| A1: Centroid (Embedder) | 0.7124 | 70.37% | 88.38% | 8.13% (65 / 800) |
| A2: Trained (Hybrid Head) | 0.7984 | 77.00% | 89.88% | 0.25% (2 / 800) |
| A3: Trained + Market Catalog | 0.7984 | 77.00% | 92.63% | 0.0000% (0 / 800) |
+-------------------------------+-----------------------+---------------------+-------------------+-----------------------+
2. Ingestion of the 300+ Market Model Pricing Catalog
The integration of our 300+ model pricing market catalog transforms the router from a static tier assigner into a real-time market cost optimizer.
Incoming Request
│
├── 1. Structural Filter ──► Image / >32k Tokens / JSON ──► Direct Fast-Path Routing
│
├── 2. Decisive Rule Precedence ──► High-Precision Regex (Math, Code, Reasoning)
│
├── 3. Local Feature Fusion MLP ──► Dense Embedding (bge-small 384-dim)
│ ├── Sparse TF-IDF N-Grams
│ ├── 12-Dim Rule Vector Features
│ └── Multi-Layer Perceptron (GELU/ReLU)
│
└── 4. 300+ Market Catalog ──► Evaluates 300+ Market Models (OpenAI, Anthropic, Gemini,
DeepSeek R1/V3, Llama 3.3/3.1, Qwen 2.5, Perplexity)
──► Serves Optimal Pareto Candidate
Key Performance Gains from 300+ Market Model Ingestion
| Performance Dimension | Legacy Keyword Router (A0) | Router v2 + 300+ Market Catalog (A3) | Operational Impact |
|---|---|---|---|
| Under-Service Rate | 10.13% (81 failures) | 0.0000% (0 failures) | 100% Elimination of Task Quality Regressions |
| Route Agreement w/ Gold Oracle | 87.25% | 92.63% | +5.38% Increase in Intent Alignment |
| Reasoning Task Under-Service | 53.66% | 0.00% | Zero Failures on Complex Reasoning |
| Math Task Under-Service | 44.58% | 0.00% | Zero Failures on Mathematical Prompts |
| Cost Regret ($/1M Requests) | -$310.02 USD | $0.00 USD | Eliminates Fake Savings from Under-Servicing |
| Cost Savings vs. GPT-4o Baseline | 95.71% | 86.62% | Saves $23,976 USD per 1M Requests |
3. Financial & FinOps Impact
Evaluating 1,000,000 requests against a standard mid-tier gpt-4o baseline demonstrates dramatic cost savings without compromising task quality:
- Flagship Baseline (
all-claude-opus-4-1): $180,139.35 USD / 1M Requests - Mid-Tier Baseline (
all-gpt-4o): $27,681.29 USD / 1M Requests - Router v2 + 300+ Market Catalog (Arm A3): $3,704.90 USD / 1M Requests
- Net FinOps Savings: 86.62% Cost Reduction ($23,976.39 USD saved / 1M requests)
Get Started in Minutes
Ready to deploy local LLM routing with 0.00% under-service risk and 86.6% cost reduction?
Mint your developer API key and start building today: