Compaction's MIN_HEADROOM_RATIO floor produces a post-compaction context cliff

[yuanhao]

Found by examples/long_horizon.rs running live against Sonnet 5 before the 0.18.0 release.

What happened

A 25-turn run over a tool returning ~60-line records, max_context_tokens: 30_000, LlmCompaction attached:

[compaction #1] Deterministic: 25 msgs / 19504 tok -> 3 msgs / 1665 tok
[compaction #2] Deterministic: 22 msgs / 21371 tok -> 1 msgs /   22 tok
[compaction #3] Deterministic: 25 msgs / 19508 tok -> 3 msgs / 1665 tok
[compaction #4] Deterministic: 25 msgs / 19892 tok -> 3 msgs / 1679 tok

Every compaction took the deterministic fallback. The briefing never landed, so the run got none of what LlmCompaction exists to provide, while still paying for the summarization requests. The agent then answered LOST when asked to recall a fact from before compaction.

Reproduced at trigger_ratio 0.6 and 0.35.

Why (partly understood)

Summarization is spawned when usage crosses trigger_ratio and spliced only when a later turn crosses the budget. With bulky tool output, the span between those two points is one or two turns — less than a summarization round-trip — so compact() finds Phase::Ready empty and falls back.

examples/llm_compaction_live.rs already documents the knob: "YO_TRIGGER … lower buys wall-clock headroom". But lowering it from 0.6 to 0.35 did not help here, which suggests the window is bounded by turns, not token fraction — one turn of a bulky tool can cross the whole gap regardless of where the trigger sits.

compact_headroom_turns exists and may be the right lever; it was left at its default in this run and is untested live.

Not explained

Compaction #2 collapsed 22 messages / 21371 tokens to 1 message / 22 tokens, far below keep_first(2) + keep_recent(6).

This did not reproduce in isolation. compaction_keeps_a_usable_well_formed_transcript (added in src/context.rs) drives the same shape through compact_messages and retains 15–24 messages / ~12.5k tokens, correctly and without orphaning tool calls. So the collapse comes from the live path — most likely LlmCompaction's own shrink_tail, or compaction running on already-compacted history — not from compact_messages.

Why it matters

Not a correctness bug: the loop always makes progress and the fallback is deliberate ("a slow or dead summarizer can never wedge it"). But a user who attaches LlmCompaction to a tool-heavy agent may pay for briefings they never receive, and silently get the lossy behaviour they were trying to avoid. CompactionMethod::Deterministic on the event is the only signal, and only if they listen for it.

Suggested work

• Establish whether compact_headroom_turns closes the gap; if it does, say so in the LlmCompaction docs next to trigger_ratio.
• Consider warning once per session when the fallback fires repeatedly — paying for briefings that never land is worth surfacing.
• Explain or fix the 22 → 1 message collapse.
• Live coverage for the splice path; every existing LlmCompaction test uses MockProvider, and none of them would have caught a fallback rate of 100%.

[yuanhao]

Root cause found — and it explains the "unexplained" collapse too

compact_headroom_turns defaults to Some(30), and effective_target_ratio computes:

let target = budget - (turns as f64) * growth_tokens_per_turn;
derived.min(self.compact_target_ratio).max(MIN_HEADROOM_RATIO)   // MIN = 0.15

"Leave room for 30 more turns" is a large demand. For any agent whose turns carry real tool output, 30 × growth_per_turn exceeds the budget outright, target goes negative, and the ratio clamps to the MIN_HEADROOM_RATIO floor.

So a tool-heavy agent runs at ~15% of its configured context budget, permanently. With max_context_tokens: 30_000 that is ~3.9k usable, which matches the observed post-compaction sizes (1665–1679 tokens after the calibrated budget is applied).

That single fact explains all three symptoms:

• Frequent compaction — 2–4 compactions in a 25-turn run. Compacting to 15% means refilling almost immediately.
• The 22 → 1 message collapse. Not a separate bug. It is the same floor applied to already-compacted history, where one bulky tool result dominates. This is also why it did not reproduce in isolation: my probe called compact_messages directly with compact_target_ratio: 0.7 and no growth measurement, so the adaptive path never engaged.
• The 100% deterministic fallback. With compaction firing constantly, the span between crossing trigger_ratio and crossing the budget is a turn or two — far shorter than a summarization round-trip. Lowering trigger_ratio cannot help, because the window is bounded by turns, not by token fraction. That is why 0.6 and 0.35 behaved identically.

Scope

Pre-existing, not a 0.18.0 regression. MIN_HEADROOM_RATIO = 0.15 and default_headroom_turns() = Some(30) are byte-identical in v0.16.5, v0.17.0 and main; git diff v0.17.0..main -- src/context.rs shows no change to this logic at all. Anyone running a tool-heavy agent on a released version has this today.

What a fix needs

This is a defaults change affecting compaction for every user, so it wants measurement rather than a patch:

• Is Some(30) right? It reads as "never let the next 30 turns overflow", but for tool-heavy work it collapses to the floor and throws away 85% of the budget. Scaling headroom to measured growth, or capping the headroom demand as a fraction of budget, both look more defensible than a fixed turn count.
• Is MIN_HEADROOM_RATIO = 0.15 the right floor once the demand is uncapped?
• Re-run examples/llm_compaction_live.rs across trigger/budget/growth combinations before changing either. The last two defaults-level flaws in this subsystem were found by simulation, not by reading, and the same applies here.
• Warn once per session when the deterministic fallback fires repeatedly — paying for briefings that never land should not be silent.

Suggest labelling this P2 and taking it after 0.18.0 rather than holding the release for a defaults change that predates it.

[yuanhao]

Measured — and the result contradicts my earlier comment on this issue

I wrote above that compact_headroom_turns: Some(30) is "the aggression that makes compaction destructive." The data does not support that. It is a deliberate trade, and the trade is real.

examples/headroom_sweep.rs drives the actual effective_target_ratio and compact_messages — not a reimplementation — across growth rates and candidate policies. Two of the four things that matter here are pure functions of the compaction code, so they sweep densely for free and deterministically, with no provider in the loop and no nondeterministic model.

At the crate-default 96K budget over 120 turns:

growth	policy	compactions	mean held	min after
tool-heavy (~3K/turn)	Some(30)	4	50,132	12,074
	Some(10)	9	70,445	63,295
	Some(5)	9	71,575	66,308
	None (flat 0.7)	9	71,575	66,308
very heavy (~8K/turn)	Some(30)	10	47,764	8,035
	Some(5)	19	66,127	48,100
	None (flat 0.7)	28	73,472	64,126

Some(30) takes half to a third as many compactions — i.e. prefix-cache invalidations — for about 30% less mean retention. That is "aggressive but rare" working as designed. Since each compaction invalidates the cached prefix past the head, fewer of them is worth real money, so the default is defensible rather than broken.

Below ~740 tok/turn at a 26K budget (~2.7K at 96K) every policy resolves identically, so none of this touches light or moderate sessions.

What the data does flag

The post-compaction cliff. Immediately after a compaction the context drops to 8–12K of a 96K budget, against 48–66K under Some(5). Mean retention recovers quickly, so it is transient — but an agent asked a question right after a compaction has very little to work with, and that is the mechanism behind the LOST answer in the earlier live run.

What is still unmeasured

The other half: do those saved invalidations actually buy cache hits? long_horizon now takes YO_HEADROOM so it can be run per policy, but the run hung twice at compaction #2 and I stopped it rather than keep spending on a flaky measurement. The offline half stands on its own; the cache half does not exist yet.

Suggested re-scope

This issue was filed as "the default is wrong." On the evidence it should become:

• Document the trade on compact_headroom_turns — the doc currently explains the mechanism but not that Some(30) buys cache stability at the cost of retention, nor that it only engages above ~2.8% of budget per turn.
• Consider the cliff — whether MIN_HEADROOM_RATIO = 0.15 should be higher, so the worst case is less severe without giving up the low compaction count. That is the one number the data actually questions.
• Measure the cache side before changing either, via YO_HEADROOM on long_horizon.

I would not change the default on what is known today.