Semantic Prompt Caching: Benchmarking Accuracy, Thresholds, and Guardrails

In LLM API gateway design, most cost-optimization levers are rate discounts: routing to a cheaper model tier, pruning context tokens, or utilizing provider prompt caching. A semantic cache hit, by contrast, short-circuits the pipeline entirely: no input tokens are billed, no output tokens are generated, and provider latency is replaced by a sub-5ms local lookup.

However, because a semantic cache returns stored completions without calling an upstream model, a false positive is a silent correctness failure.

To establish precise safety boundaries, we conducted an 800-prompt empirical evaluation of ACE's local semantic cache. This post outlines why global cosine similarity thresholds alone fail on technical prompt variants, how we engineered a hybrid 0.95 threshold + numeric guardrail architecture, and our engineering roadmap for adaptive prompt caching.

---

The Inversion Problem: Cosine Similarity vs. Semantic Meaning

The primary challenge in semantic prompt caching is distinguishing between genuine paraphrases and parameter swaps:

Pair A (Parameter Swap — Different Intent):
  Stored:   "Set statement timeout on the reporting role to 30 seconds."
  Incoming: "Set statement timeout on the reporting role to 300 seconds."
  Cosine Similarity: 0.9828  ──► Served by 0.92 threshold (INCORRECT)

Pair B (Semantic Paraphrase — Same Intent):
  Stored:   "Nightly ETL job failing with a Postgres statement timeout on reporting."
  Incoming: "A Postgres statement timeout is killing our nightly data load."
  Cosine Similarity: 0.9142  ──► Missed by 0.92 threshold (INCORRECT)

Because vector embeddings measure overall lexical and topical overlap, Pair A (a version change) scores higher cosine similarity than Pair B (a genuine paraphrase).

Adjusting a static similarity dial cannot resolve this inversion: raising the threshold above 0.98 suppresses version swaps but rejects virtually all genuine paraphrases, collapsing the system back into an expensive exact-match cache.

---

Gateway Cache Architecture

To deliver sub-5ms lookups without sending prompt text to secondary vendors, ACE runs the embedding pass in-process on CPU via ONNX runtime:

Incoming Request ──► Local ONNX Pass (bge-small-en-v1.5, 384-dim)
                         │
                         ├── Cosine Similarity Check (Threshold >= 0.95)
                         │
                         ├── Lexical Numeric Guardrail (Exact digit match)
                         │
                         └── Namespace Isolation (Tenant-bound vector partition)
                                 │
                                 ├── HIT  ──► Return Stored Completion (0ms Model Time, $0 Cost)
                                 └── MISS ──► Fall Through to Model Router & Populate Cache

Technical Specifications

• Embedding Model: BAAI/bge-small-en-v1.5 (33.4M parameters, 384 dimensions, 512-token limit, MIT licensed).
• Execution Runtime: Local CPU execution via fastembed on ONNX, baked into the gateway container image.
• Latency Profile: 2.71 ms p50 to embed a prompt, 0.29 ms flat-index scan time across 800 entries (3.26 ms p50 end-to-end hit path).
• Tenant Isolation: Every vector lookup is strictly partitioned by tenant ID, preventing cross-tenant data leakage by construction.

Compared to provider prompt caching (which requires minimum prefix lengths of 512–4096 tokens and only discounts input tokens), semantic caching works on prompts of any length and saves 100% of both input and output costs.

---

Benchmark Methodology & Dataset

We evaluated the cache across 800 prompts (420 positive pairs, 380 negative pairs) grouped into three evaluation slices:

Evaluation Design

Unlike standard pairwise benchmarks, all 800 prompts were loaded into a single shared vector namespace. Each lookup query competed against 799 distractors, testing the cache's resilience against wrong-entry hits.

---

Empirical Benchmark Results

Baseline vs. Guardrail Evaluation

Evaluating the initial 0.92 default against our upgraded 0.95 threshold + numeric guardrail system yielded the following performance across dataset slices:

Analysis: Adding the numeric guardrail (refusing cache hits if numeric literals in the prompt differ) eliminated 16–19 false hits while preserving virtually all true positive hits. On real-world agent traffic, the error rate dropped from 27.5% down to 6.5%, while precision improved to 80.3%.

---

Industry Architectural Comparison

As highlighted in recent research (vCache: Verified Semantic Prompt Caching, Schroeder et al., ICLR 2026), fixed global thresholds assume uniform embedding density across all prompt domains. Implementing adaptive per-entry thresholds represents the logical evolution for high-precision caching.

---

Financial Economics & Shadow Mode Validation

Calculating the financial return of semantic caching requires evaluating the net cost equation:

$$\text{Net Savings} = \sum_{\text{hits}} \text{Cost}(\text{Input} + \text{Output}) - \sum_{\text{requests}} \text{Cost}(\text{Local Embedding Pass})$$

Because local ONNX embeddings incur microsecond CPU overhead without per-token API charges, embedding costs remain negligible compared to model generation fees.

Risk-Free Shadow Mode

Cache hit rates are intrinsically tied to application traffic patterns. To allow engineering teams to quantify prospective savings without risk, ACE provides Shadow Mode:

• Incoming prompts are embedded and evaluated against the vector index asynchronously.
• Latency and potential cache hit rates are logged in telemetry (x-ace-cache-similarity).
• No responses are served from cache until verified by the operator.

---

Engineering Roadmap

We are advancing ACE's semantic cache through the following planned developments:

┌──────────────────────────────────────────────────────────────────────────────────┐
│                         ACE Semantic Cache Engineering Roadmap                   │
└──────────────────────────────────────────────────────────────────────────────────┘
  1. Adaptive Per-Entry Thresholds ──► Online error-bounded delta tuning (vCache model).
  2. Structural Lexical Guardrails ──► Token importance checks (negations & entity validation).
  3. Artifact Revision Pinning    ──► Version-locked fastembed ONNX model weights.
  4. Dynamic Eviction Policies    ──► Frequency & recency aware cache admission filters.
  5. Telemetry Feedback Loops     ──► Production headers (x-ace-cache-similarity) for auditability.

1. Adaptive Per-Prompt Thresholds ($\delta$-Tuning): Moving from static global thresholds to online learned per-entry boundaries calibrated against user-specified error bounds ($\delta$).
2. Entity & Negation Guardrails: Extending lexical checks beyond numbers to validate named entities and negation operators (not, never, no).
3. Embedder Revision Pinning: Locking ONNX artifact hashes in fastembed to prevent index drift across gateway image deployments.
4. Frequency-Aware Cache Admission: Implementing LFU/LRU admission policies to prevent single-use prompts from polluting vector indexes.
5. Production False-Hit Telemetry: Surfacing cache similarity scores (x-ace-cache-similarity) across response headers to enable downstream monitoring.

---

Summary

By combining local ONNX embeddings with deterministic numeric guardrails, ACE's semantic cache provides a high-precision perimeter layer that eliminates redundant model execution.

For detailed evaluation scripts and benchmark datasets, refer to skills/semantic_cache/ in the engine codebase.

---

References

1. BAAI. bge-small-en-v1.5 Model Card. huggingface.co/BAAI/bge-small-en-v1.5
2. S. Xiao, Z. Liu, P. Zhang, N. Muennighoff et al. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597
3. L. G. Schroeder, A. Desai, A. Cuadron et al. vCache: Verified Semantic Prompt Caching. ICLR 2026. arXiv:2502.03771
4. W. Gill et al. MeanCache: User-Centric Semantic Caching for LLM Web Services. arXiv:2403.02694
5. Y. Zhang, J. Baldridge, L. He. PAWS: Paraphrase Adversaries from Word Scrambling. NAACL 2019. arXiv:1904.01130