Semantic Prompt Caching: Slashing Latency and API Costs on Repeated Queries
How ACE's semantic cache short-circuits repeat and near-duplicate prompts before they reach a model — saving 100% of input and output token costs with sub-5ms local lookups.
Benchmark Update (2026-08-05). The gateway default threshold has been updated to 0.95 (from 0.92) paired with a deterministic numeric guardrail to block version/parameter swaps. For full empirical benchmark evaluation details across 800 prompts, see Semantic Prompt Caching: Benchmarking Accuracy, Thresholds, and Guardrails.
Most cost levers in an LLM gateway are rate discounts: a cheaper model, a shorter prompt, a cheaper instance. A semantic cache hit is not a discount — it removes the call. No tokens go to the model, none come back, and the cost of that call is avoided in full rather than reduced.
The mechanism addresses paraphrase. "How do I reset my password?", "how do i reset my password" and "What are the steps to reset a password?" are three distinct strings and three misses in an exact-match cache. ACE's semantic cache embeds the prompt into a vector and asks whether a sufficiently close prompt has already been answered for the same tenant. If so, it returns the stored answer without opening a connection to a provider.
How it works
- Embed the incoming prompt into a vector.
- Cosine-match it against previously stored prompt-and-completion pairs, within the same tenant's namespace only.
- If the best match clears a similarity threshold (
default 0.92— 0.95 since 2026-08-05), and its numeric literals match the stored prompt's, return the cached completion. Otherwise fall through to a model call and store the new pair.
The lookup runs before any other decision in the request path — before model routing, before prompt compaction, before any byte reaches a provider. A hit short-circuits the pipeline; a miss falls through and populates the cache after the response is returned, off the caller's critical path.
The threshold
0.92 is a configurable dial, and its setting determines which error the cache is exposed to:
- Too low and the cache answers questions that are not the same question — a correctness fault that presents as a performance gain.
- Too high and the cache fires only on byte-identical repeats, which forgoes the savings available on paraphrased traffic.
Observed similarity scores: a case or punctuation variant of a prompt scores
around 0.98, a close paraphrase around 0.99, a loose paraphrase around 0.89,
and an unrelated prompt around 0.32. At the default 0.92, near-duplicates and
typo/case variants match while loose paraphrases are rejected.
Corrected 2026-08-05. Those four figures were hand-picked examples, not a measurement. On 800 prompts the medians are 0.953 for positives against 0.922 for negatives on agent traffic — the classes overlap, so the second sentence was wrong in the direction that costs money: at 0.92 the cache rejected 83% of genuine paraphrases and served 95% of version and parameter swaps, because cosine tracks lexical overlap and "30 seconds" versus "300 seconds" scores 0.9828 while a real paraphrase scores 0.9142. That is why the fix was a numeric guard rather than a higher dial. Full numbers: Semantic Prompt Caching: Benchmarking Accuracy, Thresholds, and Guardrails. The threshold remains configurable per route.
Tenant isolation
Isolation is enforced structurally rather than by convention: every vector is written into a namespace derived from the tenant's identity, and a lookup only compares vectors within that namespace. There is no code path by which one tenant's lookup can reach another tenant's entries.
Accounting for a hit
A cache hit's "tokens saved" is the avoided call's real token count — what that exact response cost in tokens when it was first generated — not a capacity estimate. Cache savings are recorded through the same accounting the rest of the gateway uses rather than a separate counter, so the two cannot diverge.
What this post simplifies
The above treats a request as a single self-contained question, which holds for stateless classification and FAQ traffic and not for all traffic. A multi-turn conversation, or an agentic/RAG pipeline carrying retrieved context, poses a harder keying problem. The tiers this implies, and where ACE's cache is going, are in Three tiers of semantic caching.
Position in the pipeline
Model routing and prompt compaction (the next posts in this series) each still incur a cost: a cheaper model, a shorter prompt. A cache hit costs $0 and near-zero latency because the model is not called. It therefore runs first, and every other cost-saving mechanism operates on the traffic the cache did not answer.
References
- Zilliz, GPTCache: A Library for Creating Semantic Cache for LLM Queries (open source). github.com/zilliztech/GPTCache
- I. Gim, G. Chen, S. Lee, et al. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. MLSys 2024. arXiv:2311.04934
- Qdrant, Multitenancy — Isolate Data with Payload-Based Partitions. qdrant.tech/documentation/guides/multiple-partitions
- BAAI, bge-small-en-v1.5 model card. huggingface.co/BAAI/bge-small-en-v1.5