Prompt Compaction v1: A Scorecard
An 827-example benchmark evaluating ACE's local ONNX LLMLingua-2 context compactor: achieving 43.2% input token reduction while maintaining 97.2% load-bearing information retention.
Prompt Compaction v1: A Scorecard
In context-heavy AI workloads—such as document Q&A, retrieval-augmented generation (RAG), and multi-turn agentic workflows—input tokens dominate API spend. In our introduction to Prompt Compaction, we outlined how intelligent, entropy-based token pruning strips low-information prose from incoming user messages before they leave the API gateway.
With Prompt Compaction v1, we conducted a comprehensive 827-prompt evaluation benchmark comparing our local, quantized ONNX transformer compressor (LLMLingua-2) against stopword heuristic baselines across 10 diverse task domains.
The results are definitive: at our default 0.50 keep-ratio, Prompt Compaction v1 delivers 43.2% net input token savings while retaining 97.2% of critical, load-bearing information. Compared to basic stopword filtering, contextual transformer scoring cuts key information loss by 2.44× ($p = 1 \times 10^{-6}$), unlocking dramatic cost reductions without compromising application reliability.
How Prompt Compaction v1 Works
Compaction operates transparently on the hot path of the API gateway. Before a request is dispatched to an upstream provider, the compressor evaluates prose informativeness while enforcing strict structural protections:
Incoming User Message
│
├── 1. Structural Protection ──► Code Fences (```), JSON ({...}), & System Instructions
│ (Preserved byte-for-byte; never scored, never dropped)
│
├── 2. Contextual Scoring ──► Local ONNX Quantized Transformer (LLMLingua-2)
│ Scores context-aware informativeness per word
│
├── 3. Top-K Token Retention ──► Retain top-scoring 50% of prose tokens
│
└── 4. Deterministic Rejoin ──► Rejoin surviving prose with protected spans
compacted message dispatched to provider
Zero-Risk Code & Schema Safety
Prompt Compaction v1 automatically partitions code blocks, JSON payloads, and system prompt instructions prior to scoring. Fenced code and structured schemas are reproduced byte-for-byte, guaranteeing that syntax structure is never corrupted during compression.
Empirical Benchmark Scorecard ($n=827$ Prompts)
We evaluated performance across three operational keep-ratios on an independent 827-prompt dataset ($n=777$ non-contaminated out-of-domain prompts).
- Retention: Percentage of derived must-keep answer terms and critical numeric literals surviving rewrite.
- Token Savings: Actual input token reduction measured via provider BPE tokenizers.
| Keep Ratio Setting | Compressor Engine | Must-Keep Retention | Real Token Savings | Prompts Losing Key Information | Statistical Significance |
|---|---|---|---|---|---|
| 0.75 (Gentle) | Stopword Heuristic | 99.9% | 20.3% | 2 / 777 | Baseline |
| 0.75 (Gentle) | Learned ONNX | 99.8% | 20.6% | 4 / 777 | Equivalent ($p = 0.69$) |
| 0.50 (Default) | Stopword Heuristic | 92.1% | 40.7% | 95 / 777 | Baseline |
| 0.50 (Default) | Learned ONNX | 97.2% | 43.2% | 39 / 777 | Decisive ($p = 1 \times 10^{-6}$) |
| 0.33 (Aggressive) | Stopword Heuristic | 60.8% | 58.5% | 426 / 777 | Baseline |
| 0.33 (Aggressive) | Learned ONNX | 87.0% | 60.5% | 156 / 777 | Decisive ($p < 1 \times 10^{-15}$) |
Key Takeaways
- The 0.50 Ratio Sweet Spot: At
ratio=0.50, the learned ONNX compactor achieves 97.2% retention while saving 43.2% of input tokens (a 1.76× compression factor). - 2.44× Quality Advantage: On 90 distinct prompts where simple stopword filtering deleted critical answer terms, contextual transformer scoring correctly preserved them.
- Conservative Operating Boundaries: Compacting down to 0.33 causes substantial information degradation across all models and is not recommended for production.
Domain Performance & Adversarial Robustness
Performance Across Workload Domains (ratio=0.50)
| Workload Domain | Benchmark Source | Heuristic Retention | Learned ONNX Retention | Quality Retention Delta |
|---|---|---|---|---|
| Dense Prose RAG | rag-prose |
95.3% | 100.0% | +4.7% |
| Factual QA & Retrieval | qa-triviaqa |
84.2% | 98.3% | +14.1% |
| Long-Context Search | qa-longbench |
87.0% | 96.4% | +9.4% |
| Multi-Hop Reasoning | qa-hotpot |
87.2% | 96.1% | +8.9% |
| Agent Tool Trajectories | agent-trajectory |
100.0% | 100.0% | 0.0% |
| Multi-Turn Chat History | chat-history |
100.0% | 100.0% | 0.0% |
Context-aware token classification shines brightest on dense prose RAG and long-context retrieval (TriviaQA +14.1%, LongBench +9.4%, HotpotQA +8.9%). On agent tool logs and conversational histories, structural span protection ensures 100.0% retention of file paths, status codes, and UUIDs.
Latency & Architecture Efficiency
Running inference 100% locally on CPU using a quantized int8 110M parameter transformer model, Prompt Compaction v1 delivers microsecond-to-millisecond execution speeds:
| Prompt Token Length | Median Latency (p50) | Latency Overhead vs Upstream LLM Call |
|---|---|---|
| $< 1,000$ tokens | 45 ms | ~0.03% overhead |
| $1,000 – 4,000$ tokens | 457 ms | ~0.30% overhead |
| $4,000 – 12,000$ tokens | 2,744 ms | ~1.80% overhead |
Byte-Stable Prompt Cache Compatibility
Prompt Caching relies on exact byte prefix matching. ACE Prompt Compaction v1 applies compression per message, ensuring that recurring prompt prefixes compact to byte-identical strings every time. This guarantees complete compatibility with upstream provider prompt caches.
Business Impact & Enterprise Financials
By trimming 43.2% of unneeded context tokens from high-volume API streams, Prompt Compaction v1 converts unused prompt capacity into direct enterprise cost savings.
┌────────────────────────────────────────────────────────┐
│ Input Token Cost Spectrum per 1B Tokens │
└───────────────────────────┬────────────────────────────┘
│
┌────────────────────────────────────┴───────────────────────────────────┐
│ │
┌────────▼────────────────────────────────┐ ┌────────────────────────▼───────────────┐
│ Mid-Tier Pricing ($3.00 / MTok Input) │ │ Flagship Pricing ($5.00 / MTok Input) │
│ Uncompressed Cost: $3,000 / 1B Tokens │ │ Uncompressed Cost: $5,000 / 1B Tokens │
└────────────────┬────────────────────────┘ └────────────────┬───────────────────────┘
│ │
│ Compacted Cost: $1,704 / 1B Tokens │ Compacted Cost: $2,840 / 1B Tokens
▼ ▼
43.2% Direct Net Cost Reduction 43.2% Direct Net Cost Reduction
($1,296 Savings per 1B Tokens) ($2,160 Savings per 1B Tokens)
Summary of Enterprise Benefits
- Direct Financial Savings: Reduces input token billing by 43.2%, saving $1,296 to $2,160+ per 1 Billion input tokens on standard commercial model tiers.
- Expanded Effective Context Window: Enables long-context workloads to fit comfortably inside standard context windows without triggering unexpected context truncation errors.
- Enterprise-Grade Safety: Delivers 97.2% load-bearing information retention with 100% syntax protection for code and JSON payloads.
Looking Ahead
Prompt Compaction v1 proves that intelligent, context-aware token pruning can eliminate nearly half of incoming API prompt costs while preserving application quality. As context windows expand and multi-agent workflows proliferate, local context optimization will remain a vital layer of modern cloud API infrastructure.
To learn more about implementing Prompt Compaction or to explore our open evaluation harnesses, visit the ACE Fleet Documentation.
References
- Y. Pan, Z. Tan, et al. LLMLingua-2: Data Distillation for Efficient Prompt Compression. ACL 2024. arXiv:2403.12968
- H. Jiang, Q. Wu, et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. arXiv:2310.05736
- K. Łajewska et al. (Amazon). Information Preservation in Context Compression for RAG. EMNLP 2025. arXiv:2503.19114