Three tiers of semantic caching — and what's coming next for ACE's cache
A semantic cache tuned for a stateless support macro is the wrong cache for a multi-turn agent, and the wrong cache again for a RAG pipeline juggling retrieved context. Here are the three tiers of semantic caching sophistication, and ACE's roadmap to support all three.
Three tiers of semantic caching — and what's coming next for ACE's cache
Our last post walked through how ACE's semantic cache saves money today: embed the request, match it against near-duplicates, skip the model entirely on a hit. That story is true and it's real. It's also, deliberately, the simplest version of a bigger problem — because "cache this request" means something very different depending on what kind of product is sending it.
A single-shot classification endpoint, a multi-turn support agent, and a RAG pipeline compacting retrieved documents on every call are not the same caching problem wearing different clothes. They're three different problems, with three different notions of "the same question," and treating them identically is how a cache goes from a cost-saver to a correctness risk. This post lays out the three tiers we think about, and where each one sits on ACE's roadmap.
Tier 1 — Single-turn caching
This is the foundational tier: every request is self-contained, and the meaning of the request lives entirely in the text of that one request. Think classification, extraction, formatting, FAQ and support-macro answers — "what's our refund window," "pull the three fields out of this ticket into JSON," "reformat this list." There's no hidden state to get wrong, because there's no state at all. Two customers asking the same self-contained question should get the same cached answer, full stop.
This is the tier that's safest to cache aggressively, and the one where a similarity threshold and a tenant-isolated namespace are the whole safety story — nothing upstream of the request matters, because nothing upstream of the request exists.
Tier 2 — Multi-turn conversational caching
The moment a product has memory — a chat interface, a support agent that references what the customer said three messages ago, an assistant that says "yes" and means something entirely different depending on what it's replying to — a cache keyed on the latest message alone stops being safe. Two different conversations can produce an identical-looking final turn ("yes," "the second one," "go ahead") that means opposite things, and a cache that can't see which conversation it's in has no way to tell those apart.
The fix at this tier is to fold conversation state into the cache key itself — not just the newest message, but a representation of the state that message is replying to, whether that's a rolling summary, a hash of recent turns, or an explicit session boundary. The cache has to know it's answering this conversation, not just this sentence.
Tier 3 — Context-compaction-aware multi-turn caching
The hardest tier combines both problems at once: an agent or RAG pipeline where the visible question can look identical across two calls while the actual context feeding the model — retrieved documents, tool outputs, injected system state — has quietly changed underneath it, and where the prompt reaching the cache may already have been compacted or pruned before the cache ever sees it. At this tier, caching correctly means being aware of both the conversation state and what got compacted out of it, so that two requests are only ever treated as "the same" when the full context behind them, not just the surface text, actually agrees.
Why this matters for deployment, not just architecture
These tiers aren't just a technical curiosity — they're a tuning and deployment decision that should track how a customer's product actually talks to the model. A support-macro bot and a long-running conversational agent shouldn't be handed the same cache configuration, and a RAG pipeline with per-call retrieved context is a different animal again. Matching the tier to the use case is exactly the kind of decision we want ACE to make well by default, not leave to trial and error.
What's next
All three tiers are on ACE's near-term roadmap, and we expect full support across single-turn, conversation-aware, and compaction-aware caching soon — with the cache automatically adapting its keying strategy to the shape of the traffic it's seeing, rather than requiring each customer to hand-tune it themselves.
References
- Zilliz, GPTCache: A Library for Creating Semantic Cache for LLM Queries (open source). github.com/zilliztech/GPTCache
- J. Williams, A. Raux, D. Ramachandran, A. Black. The Dialogue State Tracking Challenge. SIGDIAL 2013 — the classic formulation of why conversational systems need explicit state, not just the latest utterance. aclanthology.org/W13-4065
- OpenAI, Text Generation — Managing Conversation State documentation — why chat APIs are stateless by design and callers (and anything sitting in front of them) must reconstruct context themselves. platform.openai.com/docs/guides/text-generation