Compaction: paying only for the tokens the model actually needs

Every provider bills you per input token, and input tokens are the part of the bill nobody watches. Everyone tracks output cost — it's the number on the invoice that jumps around — but the prompt you send is pure cost with no offsetting product value beyond "the model read it." And most prompts, honestly, are bloated: long chat histories that repeat themselves, filler words, a system prompt that gets nothing back for existing.

That's a cost problem you can attack before the model ever runs, which makes it cheaper than almost any other lever in the stack — you're not trading latency or quality for savings, you're just not paying to transmit words the model was never going to lean on.

What compaction does

ACE's gateway looks at every prompt before it's sent and separates it into two kinds of content: the parts that must survive untouched — code, structured data, explicit instructions — and everything else, which is negotiable prose. The protected parts are carried through exactly as written, byte for byte, no exceptions. The negotiable prose gets ranked word by word for how much meaning it's actually carrying: filler and hedging go first, repeated phrasing next, and the content that's doing real work is kept. The result is a prompt that reads leaner without losing the parts a model or a downstream parser actually depends on.

Today that ranking runs as a fast, dependency-free pass with no model inference in the loop — which keeps the added latency close to zero. The same interface is built to host a fully learned compressor down the line, one that scores informativeness the way a fine-tuned model would rather than a heuristic. That's the direction this layer is headed: same guarantee — nothing structural ever gets touched — with a steadily smarter idea of what's safe to cut.

Where it fits in the pipeline

Compaction runs after the gateway's cache check and before a request goes anywhere upstream. That ordering is deliberate: a cache hit never needs shrinking, because there's no upstream call left to make cheaper — and every request that does go upstream, streamed or not, goes out already pruned. The savings show up per request, with a clear before-and-after view of what the client sent versus what actually reached the model, so nothing about the reduction is a black box.

Not the same thing as prompt caching

Worth being precise about, because the names invite confusion: this is not the prefix-caching feature some providers offer, where an exact prefix is cached server-side and skipped on a repeat call. That mechanism only helps when the same prefix repeats byte-for-byte. Compaction rewrites the prompt itself to be shorter, which helps even when nothing repeats at all. The two are complementary, not competing, and a gateway that does both gets savings neither one reaches alone.

The takeaway

The cheapest token is the one you never send. Compaction doesn't need a model, a GPU, or a learned scorer to start paying for itself on verbose prompts — it needs a system that knows what's sacred and a sense of what's filler. The learned version of that sense is next on the roadmap, sitting behind the exact same guarantee: structure is never touched, only the padding around it.

---

References

1. H. Jiang, Q. Wu, X. Luo, et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. 2024. arXiv:2403.12968
2. T. Ge, J. Hu, X. Wang, et al. In-context Autoencoder for Context Compression in a Large Language Model. 2023. arXiv:2307.06945
3. Anthropic. Prompt Caching documentation. docs.anthropic.com/en/docs/build-with-claude/prompt-caching