The cheapest model call is the one you never make

Every dollar-saving trick in an LLM gateway is really just a rate discount — a cheaper model, a smaller prompt, a spot instance. Except one. A semantic cache hit doesn't discount the call. It skips the call entirely. No tokens go to the model, no tokens come back, and the money that call would have cost is 100% avoided, not shaved. That's a different category of savings, and it's worth treating as one.

The idea is simple: a huge fraction of production LLM traffic is the same question asked slightly differently. "How do I reset my password?" and "how do i reset my password" and "What are the steps to reset a password?" are three different strings and, to a naive exact-match cache, three cache misses. ACE's semantic cache instead embeds the prompt into a vector and asks: have we already answered something this close to it, for this exact tenant? If yes, it returns the stored answer instantly, for $0.00, without ever opening a connection to a model provider.

How it works

The mechanism is deliberately small:

1. Embed the incoming prompt into a vector.
2. Cosine-match it against previously stored prompt-and-completion pairs — but only within the same tenant's namespace, never across tenants.
3. If the best match clears a similarity threshold (default 0.92), return the cached completion. Otherwise, fall through to a real model call, then store the new pair for next time.

The lookup happens before any other decision in the request path — before model routing, before prompt compaction, before a single byte goes to a provider. A hit short-circuits the whole pipeline and returns instantly; a miss falls through and quietly populates the cache afterward, without ever slowing down the response the caller is waiting on.

The threshold is the whole product

0.92 is not a magic number, it's a dial, and which way you turn it changes what kind of mistake you're willing to make:

• Too loose (threshold too low) and the cache starts answering questions that aren't quite the same question — a real correctness bug wearing a performance-win costume. Nobody notices until a customer does.
• Too strict (threshold too high) and you get a cache that only ever fires on byte-identical repeats — technically safe, but you leave most of the savings on the table, because real traffic is paraphrased traffic.

In practice, a case/punctuation variant of a prompt scores around 0.98, a genuine close paraphrase around 0.99, a loose paraphrase around 0.89, and an unrelated prompt around 0.32. At the default 0.92, that catches near-duplicates and typo/case variants while correctly rejecting loose paraphrases — and the boundary is configurable per route, because "how loose is too loose" is a product decision, not an engineering one.

Tenant isolation isn't a nice-to-have, it's the security baseline

A cache that leaks org A's cached answer to org B isn't a performance bug, it's a data breach with extra steps. So isolation is enforced structurally, not by convention: every vector is written into a namespace derived from the tenant's identity, and a lookup only ever compares vectors within that namespace. There is no path through the system that lets one tenant's lookup see another tenant's entries — the namespace boundary is the only thing a lookup can see, by construction, not by policy.

Getting savings numbers right matters as much as getting hits right

A cache hit's "tokens saved" has to be the avoided call's real token count — what that exact response actually cost in tokens the first time it was generated — never a rough capacity estimate. A savings number that quietly overstates itself is worse than one that understates: it looks great on a dashboard right up until someone checks the math. That's why cache savings are wired straight into the same accounting the rest of the gateway uses, rather than keeping a separate number of their own.

One thing this post simplifies

Everything above treats "the request" as a single, self-contained question — which is exactly right for a lot of traffic and not the whole story for all of it. A multi-turn conversation or an agentic/RAG pipeline juggling retrieved context asks a harder question of a semantic cache than a stateless classification or FAQ endpoint does. We go into the tiers of caching sophistication this implies — and where ACE is taking the cache next — in Three tiers of semantic caching.

Why this is the first lever, not one of many

Model routing and prompt compaction (the next posts in this series) both still pay something — a cheaper model, a shorter prompt. The cache is the one layer where the answer is $0 and near-zero latency, because the model is never in the loop at all. That's why it sits first on the request path: every other cost-saving mechanism only gets to run on what the cache didn't already answer.

---

References

1. Zilliz, GPTCache: A Library for Creating Semantic Cache for LLM Queries (open source). github.com/zilliztech/GPTCache
2. I. Gim, G. Chen, S. Lee, et al. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. MLSys 2024. arXiv:2311.04934
3. Qdrant, Multitenancy — Isolate Data with Payload-Based Partitions. qdrant.tech/documentation/guides/multiple-partitions
4. BAAI, bge-small-en-v1.5 model card. huggingface.co/BAAI/bge-small-en-v1.5