LLM Router Out-of-Distribution Benchmark: Evaluating Classifier Transfer and Cost Regret

In an earlier post (LLM Router v1: a scorecard), we published the baseline metrics for ACE's difficulty-aware request router, reporting a 0.975 held-out macro-F1 score on its task classifier. While that figure accurately measures held-out accuracy on the classifier's seed distribution, production gateway traffic frequently deviates from training data distributions.

To quantify how the routing architecture generalizes to unseen workloads, we evaluated the classification pipeline and its production fallback mechanisms against an 800-prompt benchmark dataset drawn from disjoint public benchmarks.

This post details the empirical out-of-distribution performance (where macro-F1 shifts to 0.61 on unseen traffic and 0.52 under fallback heuristics), isolates the primary failure modes in high-complexity categories like math and reasoning, and outlines our engineering roadmap for adaptive, evidence-based model routing.

---

Architecture of the ACE Gateway Router

The ACE router sits on the hot path of every incoming API call, evaluating request intent and structural constraints before any tokens are generated. To avoid adding network latency or recursive API costs, the classification engine runs entirely local on CPU with zero external LLM calls.

Incoming Request
  │
  ├── 1. Structural Filter ──► Image / >32k Tokens / JSON ──► Direct Fast-Path Routing
  │
  ├── 2. Local ONNX Pass   ──► bge-small-en-v1.5 (384-dim)
  │                            ├── Category Head (Multinomial Logistic)
  │                            └── Complexity Head (Ridge Regression)
  │
  └── 3. Policy Execution  ──► Market Catalog Evaluation ──► Optimal Model Selection

Core Components

As demonstrated by Casanueva et al. (ACL 2020), pairing a frozen sentence encoder with lightweight task heads achieves competitive accuracy (93.36 on BANKING77) compared to full model fine-tuning (93.66) while drastically reducing inference overhead. However, evaluating static classifiers under distribution shift remains critical for real-world API gateway reliability.

---

Evaluated Router Configurations

Depending on environment variables and model artifact availability, the gateway executes one of four routing configurations:

Telemetry and Inspection

The active routing configuration for any request is surfaced in telemetry headers and audit logs:

• Header: x-ace-route-reason
• API Endpoint: POST /api/v1/route/explain
• Reason Identifiers: heuristic indicates arm A0, embedding centroid indicates A1, and model:0.87 (showing confidence score) indicates A2/A3.

Note on Lexical Rules: Log entries marked lexical:code indicate that an explicit code-fence pattern was detected. Both the keyword heuristic (A0) and the trained head (A2 fallback) emit this identifier.

---

Out-of-Distribution Benchmark Corpus

To evaluate transferability, we constructed an 800-prompt evaluation dataset engineered to be strictly disjoint from the router's original 798-prompt training corpus. Where training utilized MBPP and GSM8K, the benchmark uses HumanEval, BigCodeBench, MATH, and AQuA-RAT.

                  ┌────────────────────────────────────────────────────────┐
                  │          800-Prompt Evaluation Dataset                 │
                  └───────────────────────────┬────────────────────────────┘
                                              │
         ┌────────────────────────────────────┼───────────────────────────────────┐
         │                                    │                                   │
┌────────▼─────────┐                 ┌────────▼─────────┐                ┌────────▼─────────┐
│ 520 Task Prompts │                 │ 120 Structural   │                │ 160 Adversarial  │
│ 8 Task Families  │                 │ Vision, OCR,     │                │ Prompts          │
│ Disjoint Source  │                 │ >32k Long-Ctx    │                │ Multi-turn, Jargon│
└──────────────────┘                 └──────────────────┘                └──────────────────┘

Dataset Structure

• 520 Category-Specific Prompts: Distributed evenly across code, math, reasoning, qa, extraction, summarization, translation, and chat.
• 120 Structural Prompts: Multimodal image prompts, OCR workloads, and long-context documents (>32k tokens validated via native tokenization).
• 160 Adversarial Edge Cases: Multi-turn conversation drift, mixed intent, JSON-formatted non-extraction prompts, code-fence noise, and multi-lingual inputs.

---

Empirical Benchmark Results

Evaluating all four configuration arms across the 800-prompt benchmark yields the following scorecard:

Key Metrics Defined

1. Route Agreement: The percentage of requests where the router selected the exact same model tier as an oracle classifier with perfect category knowledge. Route agreement remains high (~85–90%) even when classification accuracy drops, because adjacent categories (e.g., qa vs. chat) often map to the same target model.
2. Under-Service Rate: The percentage of requests routed to a model that falls below the required capability floor for the request's true complexity. This represents a correctness risk where a user receives a cheaper model incapable of resolving the prompt.
3. Cost Regret: The signed dollar delta per 1M requests between actual router spend and oracle spend. Negative cost regret indicates under-spending driven by under-servicing complex tasks.

---

Deep Dive: Fallback Dynamics & Category Distribution

Trained Head Calibration vs. Fallback Drift

Analyzing the stage-by-stage breakdown of the trained head (Arm A2) reveals that the model itself maintains strong calibration on familiar inputs:

When out-of-distribution prompts yield low classifier confidence, the router falls back to keyword heuristics. The low accuracy of the fallback stage (24.2%) accounts for the performance gap between in-distribution seed evals and out-of-distribution benchmarks.

Concentration of Under-Service Risk

Under-service risk is not evenly distributed. Across 8 of the 11 category/complexity combinations, under-service is 0.0%. It is concentrated in complex reasoning and mathematical prompts on heuristic arms:

Enabling the trained ONNX head (Arm A2/A3) reduces math and reasoning under-service from >51% down to ~0%, eliminating the primary correctness risk.

---

Latency Profile and Cost Economics

Microsecond Classification Latency

Evaluating classification latency over single-thread CPU execution (Apple M5, batch size 1) demonstrates that local ONNX routing introduces negligible overhead compared to model generation times:

Routing Cost Reductions & Boundaries

When evaluating cost reduction against a single flagship baseline (claude-3-5-sonnet / claude-opus-4), the router achieves an 86.7% to 95.4% cost reduction across multi-vendor model catalogs.

Single-Vendor Price Floor (e.g. Anthropic Sonnet vs Haiku):   Max ~80.0% Savings
Cross-Vendor Router Selection (Sonnet vs DeepSeek / Mini):      Up to 95.4% Savings

• Single-Vendor Constraint: Routing exclusively within one provider's tier (e.g. Claude Opus to Claude Haiku) has a theoretical mathematical ceiling of ~80% due to fixed input/output price ratios.
• Cross-Vendor Optimization: Achieving >85% cost savings requires cross-provider routing (e.g., offloading structural extraction and classification to high-throughput open-weights or lightweight API models like gpt-4o-mini or deepseek-v3).

---

Product Roadmap & Ongoing Developments

To systematically address out-of-distribution transfer and improve routing fidelity, we are implementing the following roadmap:

┌──────────────────────────────────────────────────────────────────────────────────┐
│                             ACE Router Engineering Roadmap                      │
└──────────────────────────────────────────────────────────────────────────────────┘
  1. Config Versioning   ──► Pin active routing arms explicitly in environment configs.
  2. Embedder Default    ──► Enable local BGE embedder pass by default across all keys.
  3. Image Artifacts     ──► Package trained ONNX heads into container builds (42x risk reduction).
  4. Heuristic QA Branch ──► Add explicit Q&A keyword patterns to fallback stage.
  5. Soft Pattern Rules  ──► Feed structural signals (JSON/Fences) as classifier priors.
  6. IRT Model Scoring   ──► Replace manual capability matrices with Item Response Theory.
  7. RouterArena Evals   ──► Measure exact decision optimality against ground-truth outputs.

1. Deterministic Configuration Versioning: Standardizing environment variables (ACE_ROUTER_ARM=trained) in repository configuration to ensure deployment parity.
2. Default Embedder Activation: Enabling local BGE embedding passes across all runtime environments, immediately improving macro-F1 by +0.056 and reducing under-service to 9.0% even on heuristic fallbacks.
3. Container-Packaged Classifier Heads: Bundling pre-trained ONNX artifacts into base container images, driving under-service rates down from 10.6% to 0.25% (a 42× reduction).
4. Enhanced Heuristic Coverage: Adding dedicated Q&A regex patterns to eliminate qa → chat misclassifications during fallbacks.
5. Bayesian Pattern Priors: Integrating structural signals (such as json_object requests or code fences) as probabilistic priors for the classifier rather than overriding structural rules.
6. Evidence-Derived Capability Matrix (IRT-Router): Replacing hand-tuned model capability scores with empirical Item Response Theory (IRT) ratings derived from benchmark evaluations (ACL 2025).
7. RouterArena Optimality Verification: Benchmarking router decisions against actual completion accuracy and cost-efficiency trade-offs (ICLR 2026).

---

Summary and Verification

By measuring classifier performance on out-of-distribution workloads, ACE Engineering ensures that gateway optimization claims remain empirically grounded. The combination of local ONNX embeddings, trained linear heads, and cross-vendor model selection delivers sub-5ms routing decisions while maintaining multi-tier quality guarantees.

For full benchmark trace logs and evaluation scripts, refer to docs/eval_docs/ in the core engine repository.

---

References

1. L. Chen, M. Zaharia, J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2305.05176
2. I. Ong, A. Almahairi, V. Wu, et al. (LMSYS). RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
3. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618
4. I. Casanueva et al. Efficient Intent Detection with Pre-trained Sentence Encoders. ACL 2020. arXiv:2004.13845