Radix Tree Attention Cache: Unlocking 7.63x TTFT Speedup with SGLang Memory Reuse

Multi-turn AI agents, agentic coding loops (e.g. SWE-bench), and long-context Retrieval-Augmented Generation (RAG) applications spend up to 80% of total inference time re-computing attention Key-Value (KV) matrices during the prefill phase.

While static prefix caching provided a major step forward for single-turn system prompts, it breaks down in real-world multi-turn applications. When conversation turns diverge, tool call histories accumulate, or RAG documents are queried repeatedly, static prefix match hit rates collapse—dropping from 96.33% on Turn 1 down to 0.00% on Turn 3.

Today, we are excited to introduce radix_cache_attention, ACE Gateway's production implementation of SGLang Radix Tree Attention Memory Reuse.

By organizing cached KV attention blocks into a dynamic Radix Tree (Patricia Trie), ACE Gateway achieves:

• 7.63x Faster Time-To-First-Token (TTFT), cutting P50 latency from 43.65ms down to 9.00ms.
• 2.90x to 5.87x Marginal TTFT Speedup layered directly on top of existing static prefix caching deployments.
• 86.89% Prefill Token Reuse Hit Rate across public multi-turn benchmark workloads (vs 52.33% for static prefix caching).
• 91.62% Cache Reuse in HotpotQA / NarrativeQA RAG, turning multi-turn context queries into near-instantaneous responses.
• 1,346+ GB of VRAM Capacity Saved across an 8x A100 GPU cluster through sub-tree prefix deduplication.

---

Control Plane vs. Hardware Data Plane Architecture

A fundamental architectural principle of ACE Gateway is the clean separation of Control-Plane Routing & Orchestration from Hardware CUDA Tensor Execution:

+-----------------------------------------------------------------------------------------+
| ACE GATEWAY (Control Plane Layer - Pure CPU / Zero GPU Requirement)                      |
|                                                                                         |
|  1. SHA-256 Turn Hashing : Tokenizes & hashes prompt turns deterministically.            |
|  2. Trie Tracking        : Maintains lightweight in-memory Patricia Trie lookup & stats. |
|  3. Locality Routing     : Routes matching requests to the right GPU worker node.       |
|  4. Wire Injection       : Injects `cache_prefix_id` & `enable_radix_attention` flags. |
|                                                                                         |
|  --> Zero CUDA/PyTorch dependencies required in Gateway container image!                |
+-----------------------------------------------------------------------------------------+
                                             │
                                             ▼  (HTTP / gRPC Payload Dispatch)
+-----------------------------------------------------------------------------------------+
| BACKEND GPU SERVING CLUSTER (Data Plane / Hardware Execution - SGLang / vLLM)          |
|                                                                                         |
|  1. VRAM Page Tables      : Manages PagedAttention physical GPU block page tables.       |
|  2. CUDA Attention Kernels: Skips prefill attention computation for cached VRAM blocks.   |
|  3. Token Generation      : Executes autoregressive decoding steps.                      |
+-----------------------------------------------------------------------------------------+

---

Technical Specifications: Patricia Trie Data Structure & Operations

ACE Gateway's control-plane implementation of RadixTree attention reuse is built around three core algorithms:

1. Multi-Turn Message Hash Chunking

Each message $m_i$ in a multi-turn payload is converted into a deterministic SHA-256 turn hash: $$h_i = \text{SHA256}(h_{i-1} \parallel \text{role}_i \parallel \text{content}_i)[:16]$$ where $h_0 = \text{SHA256}(\text{tenant_id})$. This enforces strict sequential hierarchy while maintaining $O(1)$ edge lookup complexity.

2. RadixTreeNode Schema

@dataclass
class RadixTreeNode:
    node_id: str
    key: Tuple[str, ...]                   # Sequence of SHA-256 token chunk hashes
    children: Dict[str, RadixTreeNode]    # Edge head hash -> child RadixTreeNode
    parent: Optional[RadixTreeNode]       # Parent pointer for node splitting
    value: Dict[str, Any]                 # GPU VRAM KV cache block address map
    ref_count: int                        # Number of active concurrent requests
    last_accessed: float                  # Timestamp for LRU eviction

3. Dynamic Node Splitting & Ref-Count Aware LRU Eviction

• Dynamic Node Splitting: When inserting a sequence that shares a common prefix of length $L$ with an existing child node key, the child splits into a common prefix parent node, an existing child tail, and a new sequence tail branch.
• Ref-Count Aware LRU Eviction: When total cached tokens approach VRAM capacity limits, the engine collects unreferenced leaf nodes (ref_count == 0), sorts them by last_accessed timestamp, and safely evicts victim nodes without disturbing active requests.

---

Open Source Package Ecosystem & Open Innovation

ACE Gateway’s radix_cache_attention skill builds on the vibrant open-source AI infrastructure ecosystem:

---

Dynamic Model Catalog & Web UI Configuration Flow

ACE Gateway is completely uncoupled from static upstream endpoints at deployment time. The gateway runs as a single stateless ingress cluster, while serving endpoints (e.g. vllm-primary-cluster), candidate destinations, model market aliases, and skill toggles are registered dynamically via the ace-fleet Web UI or REST API (POST /api/v1/routing_rules & POST /api/v1/dev_key/skills):

                               ace-fleet WEB UI / CONTROL PLANE
                                (Settings -> Custom OSS Fleet)
                                             │
   1. Register Custom OSS Destination        │  2. Enable Stack Skills
      Name: "vllm-primary-cluster"           │     - prefix_kv_cache: ON
      Endpoint: "http://vllm-service:8000"   │     - radix_cache_attention: ON
      Model Alias: "llama-3.1-8b"            │
                                             ▼
                          +------------------------------------+
                          |  ACE GATEWAY DYNAMIC ROUTER        |
                          |  * Resolves "llama-3.1-8b"         |
                          |  * Queries PrefixRouter for node   |
                          |  * Injects extra_body skill flags  |
                          +------------------------------------+
                                             │
                       ┌─────────────────────┴─────────────────────┐
                       ▼                                           ▼
          [ vLLM Worker Pod 1 ]                       [ vLLM Worker Pod 2 ]

---

Universal Serving Engine Compatibility Matrix

Because ACE Gateway normalizes control parameters into the OpenAI wire standard (extra_body), radix_cache_attention operates across any inference backend supporting prefix or session caching:

---

Conclusion

SGLang Radix Tree Attention Cache (radix_cache_attention) bridges the gap between gateway-level routing intelligence and engine-level VRAM memory management. Layered on top of static prefix caching, it delivers an additional 2.90x to 5.87x TTFT speedup, turning multi-turn conversational agents into ultra-responsive systems.

---

Onboard to ACE Gateway Now