Multi-Tier Semantic Caching: Indexing Strategies for Sub-10ms Cache Hits
Categorizing semantic cache architectures across single-turn, multi-turn conversational, and context-compaction-aware tiers, and what ACE's cache key covers at each.
Reference docs: Semantic cache skill
The previous post described how ACE's semantic cache works today: embed the request, match it against near-duplicates, skip the model on a hit. That is the simplest case, because what counts as "the same request" depends on the product sending it.
A single-shot classification endpoint, a multi-turn support agent, and a RAG pipeline compacting retrieved documents on every call use three different definitions of "the same question". Applying one keying strategy to all three converts a cost-saving layer into a correctness risk. This post sets out the three tiers and what ACE's key covers at each.
Tier 1 — Single-turn caching
Every request is self-contained: its meaning lives entirely in the text of that request. Classification, extraction, formatting, FAQ and support-macro answers fall here. There is no prior state to key on, so two callers asking the same self-contained question can receive the same cached answer.
At this tier a similarity threshold and a tenant-isolated namespace are most of the safety story, because nothing upstream of the request exists. Not quite all of it: the same question under a different system prompt, tool list, tool_choice or temperature is a different request — "answer in French" and "answer as JSON" do not share an answer — so ACE's key carries an exact boundary over those four under every configuration. A request that sends none of them is keyed on the query alone.
Tier 2 — Multi-turn conversational caching
Once a product carries conversation state, a cache keyed on the latest message alone is unsafe. Two different conversations can produce identical final turns — "yes", "the second one", "go ahead" — that resolve to different answers, and a cache that cannot see which conversation it is in cannot distinguish them.
The fix is to fold conversation state into the cache key: not only the newest message, but a representation of the state it replies to — a rolling summary, a hash of recent turns, or an explicit session boundary.
This is what ACE does, by default. The query — the last user turn — is the part that gets embedded and fuzzy-matched. Everything conversational before it is rendered into a deterministic, lossy summary (user and assistant prose, in order; NFC-normalised, casefolded, whitespace-collapsed, each turn cut to its first 240 characters) and hashed into the key as an exact boundary. Three properties make that the right shape:
- Lossy where it should be. A retyped space or a capital letter in turn one must not move turn five's key. A different question in turn one must. The summary is a boundary, not a fingerprint.
- Deterministic. Not an LLM-written summary: a model call per request on the hot path of a cache is the cost the cache exists to avoid, and a non-deterministic summary is not a key.
- It does not orphan sessions. A first turn has no history and is keyed on the query alone. A repeated conversation hits turn after turn, because once turn one is served from cache, turn two's history is byte-identical to the one that wrote it.
What it gives up is the cross-conversation hit on a mid-conversation turn. On a 200-session agent corpus that hit was the residual 0.3% of served-wrong answers that survived even a caller-declared partition; the "last 2 turns" and "whole conversation" rows of that measurement are the same 0.00%, at ~11 points less hit rate than keying on the query alone. That is the trade, and it is the default rather than a per-customer setting.
Tier 3 — Context-compaction-aware multi-turn caching
This tier combines both problems. In an agent or RAG pipeline the visible question can be identical across two calls while the context feeding the model — retrieved documents, tool outputs, injected system state — has changed, and the prompt reaching the cache may already have been compacted or pruned. Caching correctly here requires keying on both the conversation state and what was compacted out of it, so that two requests match only when the full context behind them agrees, not just the surface text.
ACE covers this in two halves. What the caller placed in the prompt — the system prompt and the tool schemas — is in the key by default (Tier 1's boundary above). What the model read during the turn — tool results, MCP results, and the identity of the MCP server that returned them — is the read_set key policy (ACE_CACHE_KEY_POLICY, a deployment setting), and strict adds a refusal to serve a stateful request that declared no x-ace-cache-partition. Those are opt-in because they trade hit rate for precision in a way that is right for agents acting on live systems and wrong for a Q&A workload; the skill reference has the measured numbers and the one question that picks between them.
Why only the third tier is a deployment decision
Tiers 1 and 2 are not configuration, because there is no traffic for which they are wrong: a request that carries no system prompt and no history is keyed on the query alone, and one that does needs the boundary. The cache selects that from the shape of the request, not from a per-customer setting.
Tier 3 is different. Folding tool results into the key narrows the namespace enough to cost real hit rate, and whether that is worth it depends on whether two identical prompts a week apart legitimately deserve different answers — which a support-macro bot and an agent writing to a live system answer differently. So it is a deployment parameter that tracks how the product talks to the model.
What the key cannot see
A pasted document that differs only past its first 240 characters, or context a caller retrieved out of band and never placed in the request. Both are the caller's to declare, and x-ace-cache-partition exists for exactly that: a token naming the system an answer belongs to and the version of the context it was given.
References
- Zilliz, GPTCache: A Library for Creating Semantic Cache for LLM Queries (open source). github.com/zilliztech/GPTCache
- J. Williams, A. Raux, D. Ramachandran, A. Black. The Dialogue State Tracking Challenge. SIGDIAL 2013 — the classic formulation of why conversational systems need explicit state, not just the latest utterance. aclanthology.org/W13-4065
- OpenAI, Text Generation — Managing Conversation State documentation — why chat APIs are stateless by design and callers (and anything sitting in front of them) must reconstruct context themselves. platform.openai.com/docs/guides/text-generation