Prompt Compaction Architecture: Reducing Context Bloat and API Expenses
Pruning redundant tokens before gateway dispatch: architectural overview of heuristic and learned compaction techniques for context optimization.
Providers bill per input token. Prompts commonly carry content that is not load-bearing: chat histories that repeat themselves, filler phrasing, system prompt text that is never referenced. That cost is incurred before the model runs, so removing it does not trade latency or output quality for savings.
What compaction does
ACE's gateway inspects every prompt before it is sent and separates it into two classes: content that must survive untouched — code, structured data, explicit instructions — and negotiable prose. Protected content is carried through byte for byte. Negotiable prose is ranked word by word for the meaning it carries: filler and hedging are removed first, then repeated phrasing, and the remainder is kept.
That ranking currently runs as a fast, dependency-free pass with no model inference in the loop, which keeps added latency close to zero. The same interface can host a learned compressor that scores informativeness with a fine-tuned model instead of a heuristic, under the same constraint that structural content is never modified.
Where it fits in the pipeline
Compaction runs after the gateway's cache check and before the upstream call. A cache hit needs no shrinking, because there is no upstream call left to make cheaper; every request that does go upstream, streamed or not, is pruned first. Savings are reported per request, with a before-and-after view of what the client sent versus what reached the model.
Distinct from prompt caching
This is not the prefix-caching feature some providers offer, where an exact prefix is cached server-side and skipped on a repeat call. Prefix caching applies only when the same prefix repeats byte-for-byte. Compaction rewrites the prompt to be shorter, which applies whether or not anything repeats. The two are complementary.
References
- H. Jiang, Q. Wu, X. Luo, et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. 2024. arXiv:2403.12968
- T. Ge, J. Hu, X. Wang, et al. In-context Autoencoder for Context Compression in a Large Language Model. 2023. arXiv:2307.06945
- Anthropic. Prompt Caching documentation. docs.anthropic.com/en/docs/build-with-claude/prompt-caching