Context Optimization Benchmark: Empirical Ranking of Caching, Compaction, and Routing

Four context optimisations were specified, ranked and scheduled: de-duplicate repeat file reads, supersede stale tool results, age out old content, truncate oversized Bash output. This post is the backtest of each one against real sessions, with the arithmetic attached.

The corpus: one developer, one machine, 36 days. 149 sessions — 59 developer sessions plus the 90 subagent runs they spawned — 14,948 tool results, 14,385 API requests, 4.38 billion prompt tokens, $3,195.19 at Anthropic list price. Every figure is reproducible by re-running the script that produced it.

The dollar figures are list-price valuations, not money paid. The developer was on a subscription for the whole window; the figures are what a BYOK or API-billed team with this workload shape would be charged. The corpus was generated under zero marginal cost, so nothing in it was shaped by price and context growth is an upper bound.

One developer is one developer. The cost structure is a property of how agentic coding works. The magnitudes are this fleet's; §10 lists what that limits.

---

1. The unit, and the ceiling

The unit throughout is token-turns — tokens × turns resident. Content in the context is re-read on every subsequent request, so this is the quantity cache-read billing is proportional to. A 10,000-token file read at turn 100 of a 1,100-turn session costs 10 million token-turns, not 10,000 tokens.

Where that occupancy sits:

Bash is the largest resident tool, ahead of Read, while the published ordering put Read work first.

Every lever below reaches only that 24.2%. System prompt, tool definitions and message history are the other 75.8% and are not recorded in a transcript.

---

2. The ranking

Standalone — each lever alone against the full corpus, so the shares overlap and must not be summed:

The spread between first and last is 923×. Rows 1, 1b and 1c are the same lever at three settings, and the gap between them is larger than every other lever combined.

---

3. Both large levers are parameterised

Each of the two largest levers has exactly one parameter, and that parameter moves its value by 5–17×. Age-out at 400 turns is worth 40% of age-out at 100 turns.

The gap between the 400-turn and 100-turn windows is $182, larger than every other lever in the analysis combined. No session in this corpus was run with content removed, so the corpus contains no measurement of what a dropped result costs in task success.

---

4. Age-out

At turn 147 of a 1,317-turn session, a whole-file Read returned 10,003 tokens. The file's name does not appear anywhere in the session after turn 198 — not in a later tool call, not in a tool result; a manual scan including assistant text and thinking blocks puts the last occurrence at turn 199. The session then ran for another 1,118 turns.

96% of that read's lifetime cost accrued after the agent had stopped referring to the file. One read in one session is worth 17× the entire read de-dup lever across 36 days.

Restricting to reads ≥2,000 tokens that go ≥200 turns past their last mention finds 11 such reads worth $12.62. That filter is strict: it requires the filename to vanish from the transcript entirely, so a file mentioned once in passing at turn 900 is excluded even though its turn-9 read was unused throughout.

Risk: HIGH and unquantified. Age-out carries 71% of the BALANCED tier's value and 88% of AGGRESSIVE. With de-dup and supersede corrected downward, the token-removal thesis rests on a single parameter whose quality cost is unmeasured.

---

5. Bash truncation: the objection, tested

Bash truncation was rated HIGH risk on this objection:

"Bash output is the signal — stack traces, test failures, grep hits. A 2 KB cut removes most of a Python traceback."

Splitting the recovered value by whether the output's head carries a diagnostic (Traceback, FAILED, assert, panic:, an error line):

Nine tenths of the value is in file dumps, grep roll-ups, diffs and linter summaries. The risk is concentrated in the other tenth and is separable two ways: retain head and tail, so a traceback's exception line and its final frame both survive; or exempt anything matching the diagnostic predicate, which retains 90% of the value.

One call from turn 4 of the same 1,317-turn session: a command printing a section of a document returned 16,756 chars ≈ 4,189 tokens and stayed in context for all 1,313 remaining turns — 5,500,157 token-turns, $2.75. A 512-token cap recovers $2.41 from that single call, 7× the entire verified read de-dup lever across the corpus.

At 512 tokens this lever is 5.8× supersede and 166× verified read de-dup, bounded in the worst case, and the only one of the four whose risk profile is measurable before deployment.

---

6. Supersede

In a 411-turn session, five byte-for-byte identical browser screenshot calls were issued at turns 140, 142, 146, 148 and 150 — a UI-verification loop of screenshot, adjust, screenshot again. Each result costs ≈1,677 tokens. Once turn 142 fires, the turn-140 screenshot depicts a page state that no longer exists, and it continues to be re-read for the remaining 269 turns.

Corpus-wide: 228 hits, $9.50, 0.30% of billed, concentrated in this one behaviour. chrome/computer is $6.38 of the $9.50, against `Read`$2.29, the other browser tools $0.36 combined, and Bash $0.17.

Risk: LOW. Identity is computed on the full tool input, so re-reading a different slice of the same file supersedes nothing, and the semantic claim — that running the identical command again makes the earlier output stale — holds for exactly the case it fires on.

A correction inside this lever. Screenshots were originally priced as text. A base64 blob charged at len(base64) / 4 bills a 1451×840 JPEG at 27,878 tokens; Anthropic bills it at roughly w·h/750 ≈ 1,625, a 17× overstatement. Because screenshots dominate this lever, that error accounts for most of its former size. It also moved the denominator for everything else: tool results are 24.2% of prompt volume, not the 40.1% previously published.

---

7. Read de-duplication: measured at $0.33

The lever was specified on one statistic: 59% of file reads are re-reads of a file already read in that session. That count reproduces at 57.1% here. The redundancy inference drawn from it does not hold.

A repeat read of a file is rarely a repeat read of the same bytes. 97.6% of the re-reads are the agent requesting a different part of a file it had already opened — new content, which cannot be collapsed to a pointer.

The cause was the fingerprint: a tool call was hashed on its file path alone. One session read a 1,119-line file at offset 1 limit 470, then offset 470 limit 450, then offset 920 limit 200 — three disjoint slices of a single walk. Scored as three reads of the same target, that walk contributed $2.89 of "de-duplication value". The largest figure in the original analysis, "73 reads of one file in one session", reproduces exactly and resolves to 73 different offsets into one large file. No identical range is re-read more than 4 times in any session in the corpus.

Keying on the whole tool input and requiring the bytes to already be resident collapses the lever:

The largest single verified hit in 36 days is five cents: one file read whole at turn 715 and again at turn 895 of a 1,128-turn session, both hashing identically, with writes landing at turns 896–898 — the agent re-reading a file immediately before editing it, the most common repeat-read shape in the corpus and the reason the staleness gate exists.

At $0.33 over 36 days the lever does not cover the per-path staleness tracking it requires, and that tracking is the correctness-critical part.

---

8. Composed tiers

Applied together, first-match-wins, so no lever double-counts another's tokens:

The SAFE token-removal tier is near-empty. De-dup + supersede were published at 6.84% of tokens; verified, they are 0.69%, worth $9.77. Every larger token-volume figure involves truncation or age-out.

One experiment gates 71–88% of the remaining value: replay a session with aged context and measure task success. Until it runs, the measured position is balanced-minus — Bash truncation at 512 tokens with head+tail retention and a diagnostic exemption ($55, LOW risk), supersede alongside it ($9), age-out held.

---

9. What this changes in the product

The ACE sidecar is a local reverse proxy that sits between a coding agent and the model provider, relays traffic unchanged, prices every request, reads existing transcripts off disk, and renders the result on a local dashboard. Its first release ships zero levers — it reports "levers": [] on its health endpoint and relays every request byte-for-byte.

The dashboard ranks levers on the reader's own corpus

The lever list used to render in a fixed order. It now scores each lever alone against the reader's own sessions over the selected date range, and orders them by measured value:

Each row carries the dollars, the share of the bill, a bar proportional to the largest lever, and the risk class. Those figures are one machine's, over a corpus that has since grown to 162 sessions and $3,315.90, which is why the panel computes rather than hardcodes. Three properties:

• Standalone, not composed. Under the tier scorecards a lever is applied first-match-wins, so its contribution depends on which levers ran before it. Scored alone the shares overlap, so the panel states that and links to the composed version rather than presenting four numbers that do not sum.
• The operating points are the conservative ones — age-out at 400 turns rather than 100, a 512-token cap rather than 256. The tooltip states the value of each further setting and the risk attached.
• Every row renders disabled. Phase 0 measures only.

The corrections were made in the instrument as well

Three fixes went into the sidecar's scanner alongside this analysis: tool calls are keyed on the whole tool input rather than the target path, so a paginated walk no longer registers as duplication; de-duplication requires a matching content digest, so redundancy is proved rather than inferred; and images are priced at w·h/750 rather than base64 length. On that machine's corpus the corrected panel puts age-out at $124.69 and truncation at$57.94, against $122.57 and $54.94 in the analysis above — the same conclusions, independently computed, on a corpus 13 sessions larger.

Roadmap ordering, before and after

The published ordering was inverted. Bash truncation is first by 166×; Read de-duplication is last by two orders of magnitude.

---

10. Limits of this measurement

• One developer, one machine, 36 days. The cost structure should generalise; the magnitudes are one fleet's. A second corpus is the outstanding requirement.
• Tool results only — 24.2% of prompt volume. Everything here is a lower bound on what context reduction could reach, and says nothing about the other 75.8%.
• Residency is modelled, not observed. A result is assumed resident from its turn to the end of the session. Compaction truncates that in real sessions and fires late; where it fires, these figures overstate residency.
• The verified de-dup floor is a floor. Byte-identity and verbatim containment are provable; semantic redundancy is not measurable from a transcript and is larger than $0.33. The subset a sidecar could suppress without risking stale content is not.
• chars / 4 for text. Adequate in aggregate, wrong for any individual result.

Every number in this post moved downward at least once, and every revision was a correction to the analysis rather than new information about the workload. The pattern was identical each time: a proxy was measured instead of the quantity itself — a file path instead of a tool call, a base64 length instead of an image, a re-read instead of a re-read of the same bytes. Per-path staleness tracking, the correctness-critical kind, would have been built for 33 cents.

---

The sidecar runs locally, uploads nothing, and works with Claude Code today. The beta is opening in batches — details and signup here. The broader cost breakdown this analysis sits inside is in What 13,000 coding-agent requests actually cost.

---

Sign up to ACE now