measure: sweep the compaction headroom policy (#150) - #158
Merged
Conversation
The measurement #150 asks for, and it contradicts what I wrote there. Two of the four things that matter are pure functions of the compaction code — how often a session compacts, and how much survives each time — so they sweep densely for free and deterministically, no provider in the loop. This drives the real effective_target_ratio and compact_messages rather than reimplementing them. Result at the crate-default 96K budget over 120 turns: tool-heavy (~3K/turn) Some(30): 4 compactions, 50132 mean held Some(5): 9 compactions, 71575 mean held very heavy (~8K/turn) Some(30): 10 compactions, 47764 mean held Some(5): 19 compactions, 66127 mean held Some(30) takes half to a third as many compactions — i.e. prefix-cache invalidations — for about 30% less mean retention. That is the aggressive-but-rare trade working, not a bug. I had called it "the aggression that makes compaction destructive"; the data does not support that framing. What the data does flag is the post-compaction cliff: immediately after a compaction the context drops to 8-12K of a 96K budget, against 48-66K under Some(5). Mean retention recovers, so the cliff is transient, but an agent asked something right after a compaction has very little to work with. Below ~740 tok/turn at a 26K budget (~2.7K at 96K) every policy behaves identically, so this only affects tool-heavy sessions. long_horizon takes YO_HEADROOM so the live half — does the lower compaction count actually buy cache hits — can be run per policy. That run hung twice and is not included; the offline half stands on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement #150 asks for. It contradicts what I wrote on that issue.
Why offline
Two of the four things that matter — how often a session compacts, and how much history survives each time — are pure functions of the compaction code. So they sweep densely for free and deterministically, with no provider in the loop and no nondeterministic model.
examples/headroom_sweep.rsdrives the realeffective_target_ratioandcompact_messages, not a reimplementation.Only prefix-cache hit rate and splice rate need live runs, and those are worth spending on a shortlist rather than a grid.
Result
At the crate-default 96K budget over 120 turns:
Some(30)takes half to a third as many compactions — prefix-cache invalidations — for about 30% less mean retention. That is the aggressive-but-rare trade working, not a bug.I had called it "the aggression that makes compaction destructive." The data does not support that framing, and I have corrected it on the issue.
Below ~740 tok/turn at a 26K budget (~2.7K at 96K) every policy resolves identically, so none of this touches light or moderate sessions.
What the data does flag
The post-compaction cliff — context drops to 8–12K of a 96K budget immediately after a compaction, against 48–66K under
Some(5). Mean retention recovers, so it is transient, but it is the mechanism behind theLOSTanswer in the earlier live run.MIN_HEADROOM_RATIO = 0.15is the one number the data actually questions.What is not here
The cache half.
long_horizonnow takesYO_HEADROOMso it can be run per policy, but that run hung twice at compaction #2 and I stopped it rather than keep paying for a flaky measurement. The offline half stands on its own; the cache half does not exist yet, and I would not change the default without it.No library change — an example plus one env knob on an existing example.
🤖 Generated with Claude Code