Multi-Tier Semantic Caching: Indexing Strategies for Sub-10ms Cache Hits
Categorizing semantic cache architectures across single-turn, multi-turn conversational, and context-compaction-aware tiers to deliver sub-10ms response times without correctness degradation.
The previous post described how ACE's semantic cache works today: embed the request, match it against near-duplicates, skip the model on a hit. That is the simplest case, because what counts as "the same request" depends on the product sending it.
A single-shot classification endpoint, a multi-turn support agent, and a RAG pipeline compacting retrieved documents on every call use three different definitions of "the same question". Applying one keying strategy to all three converts a cost-saving layer into a correctness risk. This post sets out the three tiers and where each sits on ACE's roadmap.
Tier 1 — Single-turn caching
Every request is self-contained: its meaning lives entirely in the text of that request. Classification, extraction, formatting, FAQ and support-macro answers fall here. There is no prior state to key on, so two callers asking the same self-contained question can receive the same cached answer.
At this tier a similarity threshold and a tenant-isolated namespace are the complete safety story, because nothing upstream of the request exists.
Tier 2 — Multi-turn conversational caching
Once a product carries conversation state, a cache keyed on the latest message alone is unsafe. Two different conversations can produce identical final turns — "yes", "the second one", "go ahead" — that resolve to different answers, and a cache that cannot see which conversation it is in cannot distinguish them.
The fix is to fold conversation state into the cache key: not only the newest message, but a representation of the state it replies to — a rolling summary, a hash of recent turns, or an explicit session boundary.
Tier 3 — Context-compaction-aware multi-turn caching
This tier combines both problems. In an agent or RAG pipeline the visible question can be identical across two calls while the context feeding the model — retrieved documents, tool outputs, injected system state — has changed, and the prompt reaching the cache may already have been compacted or pruned. Caching correctly here requires keying on both the conversation state and what was compacted out of it, so that two requests match only when the full context behind them agrees, not just the surface text.
Why the tier is a deployment decision
The tier is a tuning and deployment parameter that should track how a given product talks to the model. A support-macro bot, a long-running conversational agent, and a RAG pipeline with per-call retrieved context need different cache configurations.
What's next
All three tiers are on ACE's near-term roadmap, with the cache selecting its keying strategy from the shape of the traffic rather than requiring per-customer tuning.
References
- Zilliz, GPTCache: A Library for Creating Semantic Cache for LLM Queries (open source). github.com/zilliztech/GPTCache
- J. Williams, A. Raux, D. Ramachandran, A. Black. The Dialogue State Tracking Challenge. SIGDIAL 2013 — the classic formulation of why conversational systems need explicit state, not just the latest utterance. aclanthology.org/W13-4065
- OpenAI, Text Generation — Managing Conversation State documentation — why chat APIs are stateless by design and callers (and anything sitting in front of them) must reconstruct context themselves. platform.openai.com/docs/guides/text-generation