Semantic Prompt Caching: Benchmarking Accuracy, Thresholds, and Guardrails

In LLM API gateway architecture, standard cost-optimization techniques focus on incremental rate discounts—such as routing to lower-tier models, trimming context windows, or relying on provider-side KV prompt caching. A semantic cache hit, by contrast, short-circuits the LLM pipeline entirely: zero input tokens billed, zero output tokens generated, and provider latency replaced by a sub-5ms local vector lookup.

However, because a semantic cache returns a pre-stored completion without invoking an upstream model, returning an incorrect cache entry represents a silent correctness failure.

To establish robust production safety boundaries without sacrificing throughput, we conducted an 800-prompt empirical benchmark of ACE's local gateway cache. This post details why single-vector cosine similarity thresholds fail on technical prompt variants, how we engineered a multi-layer SemanticPairVerifier post-retrieval guard system, and the quantitative business impact of serving 73.88% of requests at a sub-10% served-wrong safety budget.

---

The Inversion Problem: Cosine Similarity vs. Semantic Intent

The foundational challenge in semantic prompt caching is that dense vector embeddings measure broad topical and lexical co-occurrence rather than strict logical equivalence. This creates a fundamental semantic inversion:

Pair A (Parameter Swap — Different Intent):
  Stored:   "Set statement timeout on the reporting role to 30 seconds."
  Incoming: "Set statement timeout on the reporting role to 300 seconds."
  Cosine Similarity: 0.9828  ──► Served by 0.95 threshold (INCORRECT / FALSE HIT)

Pair B (Semantic Paraphrase — Same Intent):
  Stored:   "Nightly ETL job failing with a Postgres statement timeout on reporting."
  Incoming: "A Postgres statement timeout is killing our nightly data load."
  Cosine Similarity: 0.9142  ──► Missed by 0.95 threshold (INCORRECT / FALSE MISS)

Because Pair A shares near-identical word choice, its cosine similarity outscores Pair B (a genuine technical paraphrase).

Static threshold tuning cannot solve this inversion. Raising the similarity cutoff to 0.99 suppresses version and parameter swaps but discards legitimate paraphrases—collapsing the system back into an expensive exact-string match cache.

---

Gateway Cache Architecture & SemanticPairVerifier

To deliver sub-5ms lookups while guaranteeing zero external data exposure, ACE executes vector embedding in-process on CPU via ONNX Runtime, paired with a deterministic post-retrieval verifier pipeline:

Incoming Prompt ──► Local ONNX Pass (bge-small-en-v1.5, 384-dim)
                         │
                         ├── Vector Index Lookup (Top-1 Candidate Entry)
                         │
                         └── SemanticPairVerifier Pipeline
                                 ├── 1. Cosine Similarity Check (Threshold >= 0.65)
                                 ├── 2. Numeric & Version Guard (Exact digit matching)
                                 ├── 3. Negation & Scope Guard (Operator parity)
                                 └── 4. Syntactic Alignment PAWS Guard (LCS & 2-Gram Overlap)
                                         │
                                         ├── HIT  ──► Return Stored Completion (0ms Model Time, $0 Cost)
                                         └── MISS ──► Forward to Model Router & Populate Cache

Four-Tiered Verification Guardrails

1. Cosine Similarity Threshold: Acts as the initial high-recall candidate retrieval filter.
2. Numeric & Version Guard: Extracts digit patterns and version identifiers (_numeric_tokens), blocking cache hits when numbers differ (e.g., 30s vs 300s).
3. Negation & Scope Guard: Evaluates asymmetric presence of negation and restriction keywords (not, no, never, disable, deny, except, without).
4. Syntactic Alignment PAWS Guard: Targets high-overlap word-scramble false positives. For pairs with $\ge 65\%$ unigram overlap, it computes the Longest Common Subsequence (LCS) ratio ($LCS / \max(|T_1|, |T_2|)$) and 2-gram Jaccard overlap, rejecting pairs where sentence structure has been scrambled.

---

Benchmark Methodology & Dataset

We benchmarked the gateway cache across 800 prompts (420 positive pairs, 380 negative pairs) spanning three dataset slices:

Shared Namespace Benchmark Protocol

Unlike standard pairwise benchmarks that score a query only against its matching pair, all 800 prompts were loaded into a single shared vector namespace. Every incoming probe competed simultaneously against 799 distractors, rigorous testing against cross-entry misallocation.

---

Quantitative Benchmark Results

Deploying the tuned SemanticPairVerifier achieved a major performance breakthrough across all metrics compared to the original un-guarded baseline ($\theta = 0.95$):

Key Analytical Takeaways

• Recall & Precision Double-Gain: True-Hit Rate jumped from 50.48% to 75.71%, while Hit Precision rose from 51.08% to 74.92%.
• Drastic Reduction in Served-Wrong Answers: Served-Wrong Rate dropped from 25.37% down to 9.62%, comfortably beating the strict $\le 10\%$ production safety ceiling.
• Microsecond Latency Overhead: The CPU-local ONNX embedding pass requires 2.71ms p50, and the SemanticPairVerifier adds $<0.5\text{ms}$ post-verification logic—maintaining a total lookup latency under 3.5ms.

---

Financial Impact & Business Value

The financial return of semantic prompt caching directly scales with avoided LLM generation token costs:

$$\text{Net Savings} = \sum_{\text{Cache Hits}} \text{Cost}(\text{Prompt} + \text{Completion Tokens}) - \sum_{\text{Requests}} \text{Cost}(\text{Local Verification})$$

Economics at Scale (Per 1 Million Requests)

By intercepting nearly 3 out of 4 incoming requests with high precision, enterprise platforms running millions of monthly agent queries recover significant gross margin while simultaneously offering sub-10ms response times to end users.

---

Forward-Looking Vision: Next-Gen Caching Infrastructure

The success of SemanticPairVerifier establishes a new blueprint for intelligent gateway perimeters. Our engineering roadmap includes:

1. Adaptive Per-Entry Thresholding ($\delta$-Tuning): Moving to online error-bounded threshold calibration that dynamically adapts cutoff boundaries based on local vector cluster density.
2. Structural Graph Verification: Incorporating AST and JSON schema validation for structured tool-use payloads to guarantee structural equivalence.
3. Real-time Telemetry & Quality Feedback: Exposing x-ace-cache-similarity and verification metrics in gateway headers for automated observability and downstream auditing.

---

Conclusion

Combining local ONNX embedding retrieval with multi-guard post-verification proves that gateways do not have to choose between high cache hit rates and strict answer accuracy. By elevating hit rates to 73.88% while suppressing served-wrong errors to 9.62%, ACE delivers an enterprise-grade semantic cache that maximizes cost efficiency while preserving absolute answer integrity.

---

References

1. BAAI. bge-small-en-v1.5 Model Card. huggingface.co/BAAI/bge-small-en-v1.5
2. S. Xiao, Z. Liu, P. Zhang, N. Muennighoff et al. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597
3. L. G. Schroeder, A. Desai, A. Cuadron et al. vCache: Verified Semantic Prompt Caching. ICLR 2026. arXiv:2502.03771
4. Y. Zhang, J. Baldridge, L. He. PAWS: Paraphrase Adversaries from Word Scrambling. NAACL 2019. arXiv:1904.01130

---

Sign up to ACE now