fix(assistant): stop chained compaction from growing the digest without bound - #154
fix(assistant): stop chained compaction from growing the digest without bound#154juacker wants to merge 1 commit into
Conversation
…ut bound Session-rotation compaction grew instead of compacting. Measured over 25 recorded compactions: the digest went 56 KB -> 960 KB across 20 rotations, adding exactly one nested boilerplate preamble each time, against a SUMMARY_TRANSCRIPT_MAX_CHARS budget of 96,000 — 10x over. Two compounding causes, both in the deterministic digest path: - select_head_and_tail never enforced its own budget. Its head loop always keeps at least one message, and message 0 of a rotation window is the *previous* digest, so each digest copied its predecessor whole and then appended the tail. The head/tail-meet early return had the same hole. - The prior digest was rendered back in with its own preambles intact while the new summary message added a fresh copy on top, so boilerplate stacked one layer per rotation. Fix: clamp any single message to the budget of the slice it lands in (head keeps the opening, tail keeps the most recent text, both noting where to recover the full text), and strip a prior summary's preambles when it is rendered into a new digest. Preambles become named constants so the strip is exact. No behaviour change when the history already fits the budget. Adds 5 tests, including a 12-rotation chaining test asserting the digest stays within budget and does not creep upward.
|
Correcting my own PR body before review: it claimed every rotated session is seeded with ~240K tokens. That was wrong — The fix still holds, for a sharper reason. The clamp keeps head 2/3 (42,666 bytes), which exists to preserve the original goal. Measured on the current live digest: real content starts at byte 21,657, with 21 of 23 nested preambles sitting inside the head. So 51% of the model-visible seed is stacked boilerplate, growing ~941 bytes per rotation — on track to consume the whole head budget at ~45 rotations. That makes |
1 participant
PR — fix(assistant): stop chained compaction from growing the digest without bound
Branch
fix/compaction-digest-growth, one file:src-tauri/src/assistant/compaction.rs.The bug
Compaction of a CLI session (
session_rotation_summary) does not compact. It grows, monotonically,by roughly the size of its own predecessor, every single rotation.
Measured over the 25 compactions recorded in this workspace's
.clai/data.sqlite:SUMMARY_TRANSCRIPT_MAX_CHARSis 96,000. The current digest is 10× its own budget, andexactly one extra copy of the boilerplate preamble is added per rotation — a perfect linear
signature, not noise. The
local_summary(model-summarised) path in the same table stays at3.6–10 KB throughout, so this is specific to the deterministic rotation path.
What this actually costs the model (corrected 2026-08-07)
An earlier draft of this section claimed the model is seeded with ~240K tokens. That was wrong,
and the real number is worth stating precisely because the fix should be justified by what it
actually buys.
The stored digest never reaches the model whole.
local_agent.rsclamps it at the CLI-sessionboundary via
truncate_cli_context_head_tail(value, CLI_FRESH_CONTEXT_SUMMARY_MAX_BYTES), withCLI_FRESH_CONTEXT_SUMMARY_MAX_BYTES = 64_000— head two-thirds (42,666 bytes) + tail one-third,joined by
[... middle of oversized compacted summary omitted ...]. So the seed is ~16K tokens,not 240K.
The harm is not volume, it is what fills the head. The head 2/3 exists to preserve the opening
of the conversation — the original goal. In the current live digest it no longer does:
# Continuation Summary) begins51% of the head budget is spent on stacked copies of the preamble before one word of content,
and the stack grows ~941 bytes per rotation. Extrapolated, the preamble stack alone swallows the
entire head budget at ~45 rotations, at which point a rotated session is seeded with boilerplate
and nothing else. The budget the code believes it is enforcing has also been fiction since
rotation 3.
This is why the fix has two parts: the clamp bounds the stored digest, and
strip_compaction_preamblesis what reclaims the head.Root cause
Two compounding defects, both in the deterministic digest path.
1.
select_head_and_taildoes not enforce its own budget. Its contract is "join renderedmessages within
budget". The head loop always keeps at least one message —if next > head_budget && head_end > 0 { break }— so a single oversized message is accepted whole.In a chained compaction, message 0 of the window is the previous digest
(
provider_history_messages_with_compactionputs it at position 0, andselect_compaction_windowdeliberately keeps compaction summaries compactable). So the "at least one" guarantee copies the
entire predecessor into every successor, then appends the tail on top. The
tail_start <= head_endearly return had the same hole.
2. Preambles re-nest. The previous digest is rendered back into the transcript with its own
summary_message_textandfallback_summarypreambles intact, and the new summary message adds afresh copy on top — so the head of the digest slowly degenerates into stacked boilerplate that
carries no information the model isn't already told at the top.
The fix
clamp_slice/clamp_message: no single message may exceed the budget of the slice it lands in.Truncate rather than drop, so the opening goal still survives — just not at any size. The head
slice keeps the start of an oversized message (the opening goal), the tail slice keeps the end
(the most recent exchange), both with an explicit note pointing at
history_queryfor the fulltext. Applied on both the omission branch and the head/tail-meet branch.
strip_compaction_preambles: when a prior compaction summary is rendered into a new digest, itsown preambles are stripped. The two preambles are now named constants (
SUMMARY_MESSAGE_PREAMBLE,FALLBACK_DIGEST_PREAMBLE) instead of inline literals, which is what makes the strip exact.No behaviour change for sessions that fit inside the budget: the
total <= budgetfast path isuntouched, and clamping is a no-op on messages that already fit.
Validation
cargo test --lib— 946 passed, 0 failed (5 new).cargo fmt --check,cargo clippy(2 warnings, both pre-existing in
config/mod.rs),npm run typecheck,npm run lint,git diff --check.New tests, each verified to fail against the pre-fix behaviour (clamp made a no-op and the
preamble strip removed — 4 of 5 fail, then pass with the fix restored):
select_head_and_tail_never_exceeds_budget_on_one_huge_message— two 500 KB messages, 4 KBbudget; asserts the output fits, and that head and tail both still survive.
select_head_and_tail_bounds_output_when_head_and_tail_meet— the branch where no middle isomitted, which had the same hole.
chained_digests_do_not_grow_without_bound— simulates 12 successive rotations, feeding eachdigest back as message 0 of the next window. Asserts the final digest is within budget and is
not creeping upward round over round. This is the regression test for the table above.
chained_digests_do_not_stack_preambles— 6 rotations, exactly one copy of each preamble.strip_compaction_preambles_leaves_ordinary_text_alone.Not covered
Digests already stored in
data.sqlitestay large; there is no migration. Any session currentlycarrying a multi-hundred-KB digest keeps it until its next rotation, which will then produce a
bounded one.
Why this matters beyond token cost
.clai/memory/knowledge.mdrecords a confirmed finding from an earlier investigation: the agentfabricates tool calls and results in exactly one place — the first turn after a rotation compaction —
and it happened only on the largest digests (450–796 KB, nesting 10–17), never on the small
local_summaryones (9–11 KB, nesting 1). 4 of 34 rotations. That correlation now has a mechanism:those digests were large because of this bug. Bounding them is the first plausible fix for the
fabrication failure, though this PR does not prove causation.