Learned prompt compaction, measured on 827 prompts

An earlier post described compaction as pruning dead weight out of a prompt before it leaves the gateway. It described two ways of deciding which words are dead weight: a word-local heuristic — a stopword and filler list, free, no model — and a learned scorer, a fine-tuned transformer that reads each word's informativeness from its neighbours.

The heuristic ships on by default. The learned scorer is a 170MB artifact and a forward pass in front of every request. Whether the second is worth the first was previously decided on six hand-annotated prompts — a sample too small to separate the two scorers from sampling noise.

This post reports the 827-prompt measurement, including the conditions under which compaction should not be used at all.

What compaction does

The input is one message from your application. The output is a shorter version of that message, which is what actually gets sent to the model provider.

your message
  -> set aside the parts that must not change
       code blocks, JSON, system instructions — copied through untouched
  -> give every remaining word a score
  -> keep the highest-scoring half, delete the rest
  -> stitch the survivors back together
shorter message -> provider (you are billed for this one)

The setting called ratio is the fraction of words kept, so ratio=0.5 keeps half. System prompts and tool definitions are never touched.

The scoring step is the only thing that differs between the two scorers. The heuristic scores each word on its own — is it a stopword, is it filler. The learned model scores each word by reading the words around it. Everything else is identical, which is what makes this a fair comparison, and it is one the research literature does not run: published benchmarks compare one learned compressor against another, never against the free option.

What a good result and a bad result look like

Here is a real prompt from the benchmark, and what each scorer did to it. The question is "What is the original meaning of the word Norman?" and the answer, sitting in the supplied context, is Viking.

A bad result. The learned model cut the sentence one word too early:

...Latinized Nortmannus, Normannus, Nordmannus Medieval 9th "Norseman,

The word Viking is gone. Notice what this failure looks like: the output is shorter, it reads fine, nothing is obviously broken — and the question can no longer be answered. That is the entire risk of compaction. It does not fail loudly.

A good result. On the same prompt the heuristic kept it:

...Nordmannus (recorded Medieval Latin, century) mean "Norseman, Viking".

And the reverse happens too. On another question whose answer was "10th century", the heuristic kept century but deleted 10th — leaving an answer that is wrong rather than missing. The learned model kept both. 10th is not a stopword and not filler, so a scorer looking at that word in isolation has no way to know it is the half that matters. A scorer reading the sentence does.

Those two examples are the whole benchmark in miniature: 827 prompts, and we count how often each scorer deletes the word that mattered.

The corpus

827 prompts across ten families, built from public benchmarks plus real agent-shaped traffic.

The constraint is deciding which words must survive. Hand-annotation does not scale and cannot guarantee the labels were fixed before the results were seen, so every must-keep term is derived by rule:

• Extractive QA (SQuAD v2, HotpotQA, LongBench, TriviaQA) — the dataset ships a gold answer, the row is kept only if that answer occurs verbatim in the context, and the must-keep set is the content words of that span. A paraphrased answer is a dropped row, not a rewritten one.
• Arithmetic (GSM8K) — every number in the question. Lose one and the problem is unsolvable. No judgement involved in deciding that 16 matters.
• Agent traffic — identifiers, regions, file paths, status codes, timestamps and negations, extracted by rule from the prompt's own text.

Then four filters remove terms that would measure nothing: terms that were never in the prompt, terms living only inside a protected code fence (retained by construction — they would inflate both scorers equally), terms reachable through a longer word (not inside cannot), and terms so common they cannot realistically be lost. Building the corpus dropped 1,072 rows and 798 candidate terms, each against a counted reason.

One family is excluded from every headline number. The learned scorer is LLMLingua-2, fine-tuned on MeetingBank. The corpus includes 50 MeetingBank rows labelled training-set, because retention there measures in-domain memorisation rather than compression skill. They are reported separately.

How the score is computed

For each prompt we run the compressor, then check which of the required words are still present.

Retention is the fraction that survived. In the Norman example: one required word (Viking), the learned scorer kept zero of it, so retention is 0.0 for that prompt. The heuristic kept it, so retention is 1.0. Average that over hundreds of prompts and you get the numbers below.

Tokens saved is measured with the same tokenizer your provider bills with, so a "43% saving" means 43% off the input line of your invoice, not 43% fewer words.

One honest limitation worth stating plainly. We label which words must survive. We do not label which words are safe to delete — doing that would mean deciding, for every word of 827 prompts, whether the answer still holds without it, and that is exactly the judgement call we removed by deriving labels from gold answers instead.

So retention only measures one kind of mistake: deleting something important. It says nothing about the opposite mistake, keeping something useless. That is what the tokens-saved column is for. A scorer that deleted nothing would score a perfect 1.0 on retention and save you nothing, so the two numbers have to be read together. Neither one alone is a grade.

Results

We ran both scorers over all 827 prompts at three compression settings. ratio is the fraction of words kept, so 0.75 is gentle and 0.33 is aggressive. Retention is the fraction of load-bearing terms that survived.

The learned scorer's value is entirely ratio-dependent.

At ratio 0.75 the two scorers are equivalent. The difference is −0.0009 retention, with a 95% confidence interval spanning zero and an exact McNemar test at p = 0.69 on 6 discordant prompts. Both retain 99.9% of must-keep terms. A deployment compressing this gently gains nothing measurable from the 170MB artifact.

At ratio 0.50, the default, the learned scorer improves retention by 0.051 (95% CI [+0.009, +0.094], McNemar p = 1×10⁻⁶) and also saves 2.5 percentage points more tokens — it is not trading quality for savings.

The averages understate what is happening. Compared prompt by prompt, at the default ratio:

Ninety rescues against thirty-four breakages. That 90-versus-34 imbalance is the actual case for running the model; the 0.051 average is just a summary of it.

At ratio 0.33 the heuristic drops a load-bearing term in 55% of prompts, against 20% for the learned scorer. Neither is suitable for production at that setting.

Where the learned scorer helps

Per family, at the default ratio:

The advantage is concentrated in retrieval — long documents where the answer is a single span and word-local scoring has no basis for judging informativeness. On agent trajectories and chat history both scorers retain everything, because the must-keep terms there are identifiers and numbers that a stopword list does not drop.

On arithmetic the heuristic is better. A stopword list retains every numeral by construction; the learned scorer scores numerals in context and occasionally rates one uninformative. It dropped a required number on 4 of 140 arithmetic prompts against 1 for the heuristic. The effect is small, but the failure mode — a deleted digit — is unrecoverable.

The excluded family, quantified

On the 50 MeetingBank prompts — the corpus the checkpoint was fine-tuned on — the learned scorer scores 1.000 and loses no terms. Its out-of-domain figure is 0.972. That difference is the reason the family is reported separately rather than averaged into the headline.

What retention does and does not prove

Retention counts whether load-bearing terms survived the rewrite. It does not count whether the model still answered correctly, and the two come apart in both directions.

Amazon's information-preservation study (Findings of EMNLP 2025) measured this: the xRAG compressor scores 0.297 exact-match on HotpotQA while retaining 13% of entities. A method can discard most named content and still post a usable task score, because the target model recovers from redundancy or parametric knowledge. Its worst-retained categories were dates and cardinal numbers, which are the categories this benchmark treats as must-keep.

A dropped term is therefore not proof of a wrong answer, and a retained term is not proof of a right one: keeping 412.6 while deleting the clause that says what it measures scores 1.0 and answers wrong. Retention is a controlled comparison between two scorers on identical prompts. It is not a task score and is not comparable to published task scores.

What this means for your traffic

Which of these numbers should you expect to see on your own prompts:

The honest gap: this benchmark counts deleted words, not wrong answers. A prompt that scored badly here might still have produced a correct response, and a prompt that scored perfectly might not have. Closing that gap means A/B testing against a live model, which we have not done.

The second gap is mix. Two thirds of this corpus is question-answering, because that is where answers can be checked automatically. Your traffic is not two thirds question-answering — and the families here that most resemble a real gateway's load are exactly the ones where the two scorers tied.

Cache safety

Prompt caching is a strict prefix match, so a compressor that rewrites the prompt can invalidate the cache it was meant to complement. Measured for ACE: it does not, and the reason is architectural. Compaction is applied per user message, never to the assembled prompt and never to the system prompt or tool definitions, so each message compacts as a pure function of its own text.

A recurring document turn, compacted across five conversations with five different following turns:

The second row is the default behaviour of a compressor applied to the assembled prompt. It would invalidate every cache hit while leaving all quality metrics unchanged.

The same property separates this design from LongLLMLingua, which is query-aware: it scores context against the question and produces a different prefix per query. An independent cost study measured that design at +40% cost versus no compression on an agent benchmark, while completing more tasks — a cost regression, not a quality trade. ACE's scorer does not receive the query.

Compaction versus prompt caching

Every frontier provider discounts cached input tokens by approximately 90%; DeepSeek discounts 98–99%. Against that discount, the measured token reduction is worth:

A cache hit is worth approximately 2.1× what compaction is worth, and the two are not additive. Compaction reduces the count of tokens billed at full rate; caching reduces the rate on tokens sent repeatedly. Applying a published two-tier cost model to the measured compression ratio puts the crossover at approximately a 59% cache hit rate — above that, compaction should be off.

This is why compaction is a per-key toggle rather than an imposed default:

• Compaction earns its keep on cache-cold traffic — one-shot requests, high-cardinality RAG context, prompts that never repeat. There is no cache hit to lose, and the reduction is real money.
• On cache-hot traffic, leave it off and let the cache do the work.

Because ACE's compaction is byte-stable, enabling it does not invalidate cache hits. It does not outperform them either.

Where compaction does not pay

Named conditions, not hedges:

1. Cache-hot traffic, above roughly a 59% hit rate. On DeepSeek's 98% hit discount, effectively any prefix reuse beats compaction.
2. Short prompts. Reducing a 500-token prompt by 43% at Opus 5 rates saves $0.001, and the scoring cost is paid regardless of how little there is to save.
3. Output-dominated workloads. Output costs 5–6× input. A request with 500 input and 4,000 output tokens sees a 43% input cut save 1.1% of the request. A pre-registered trial also measured aggressive compression increasing output length.
4. Payloads whose value is exact reproduction — JSON, SQL, code, legal text. Deleting one token from valid JSON produces invalid JSON: a failure with no gradient. The compressor protects fenced code and parseable JSON spans byte-for-byte, which is why this is a caveat about unfenced payloads rather than a defect.
5. Very long contexts. Our learned scorer covers about 12k tokens of prompt; past that the tail falls back to the word-local heuristic. Peer-reviewed work finds compression matters most in long contexts — so the model stops exactly where the stakes rise.

Compression ratio

The default keeps 50% of words, a measured 1.76× compression — below the 2×–5× range in the literature. A pre-registered randomized trial over 358 Claude Sonnet 4.5 runs measured a 0.5 keep-rate at −27.9% total cost and a 0.2 keep-rate at +1.8%, dominated on both cost and output similarity, with compressed prompts producing longer outputs. Its stated conclusion: "'compress more' is unreliable in production settings." The measurements above are consistent with that: at 0.33 both scorers drop load-bearing terms at rates unsuitable for production.

Reproducing this

The corpus builder, the eval harness and the cache-stability check are in the ACE repository. The benchmark makes no provider API calls and is deterministic: same seed, same rows, same digest. Retention and token savings are hardware-independent.

The full analysis — per-family breakdown, paired statistics, drop accounting, and comparability verdicts against each published number cited here — accompanies the machine-readable results.