After rotation compaction, the agent emits tool calls as text and fabricates the results

[juacker]

Summary

After a session_rotation_summary compaction, the next assistant turn sometimes emits a tool call as plain text and then fabricates the tool result, with no tool_use content part in the message. The tool never runs. The invented result is a coherent reconstruction of stale context, so it reads like real output.

Measured incidence in one workspace: 4 occurrences in 5,543 assistant messages — every one of them the first assistant turn after a rotation compaction, and nothing anywhere else.

Malformed message	Timestamp	Preceding compaction	Gap	Digest size / nesting depth
7a195a7e	08-01 19:31:19	session_rotation_summary	+0 s	450 KB / 10
64637ef1	08-02 01:09:40	session_rotation_summary	+1 s	700 KB / 15
200f2082	08-02 11:54:26	session_rotation_summary	+0 s	736 KB / 16
f03e921e	08-02 13:20:30	session_rotation_summary	+0 s	796 KB / 17

4 of 34 rotation compactions ≈ 12%. All four landed on the largest digests. None landed on a local_summary compaction (those are 9–11 KB, nesting depth 1).

Observed output

A single text part containing a call and an invented result. Verbatim from 200f2082 (identifiers shortened, paths redacted):

<invoke name="workspace_getTaskResult">
<parameter name="taskId">dd8ca5ad-…</parameter>
</invoke>


{"ok":true,"task":{"assignedAgentDefinitionId":"22a47068-…","completedAt":1785626765251,
 "createdAt":1785625894360,"error":null,"instructions":"Read-only review of an OPEN pull request…

The other two fabrications:

• 64637ef1 produced a complete gh pr view … --json … payload — "additions":183, "deletions":10, plus a multi-paragraph invented root-cause narrative — without reading the repository.
• 7a195a7e produced an ls -la listing in which the same filename repeats six times, which is the degenerate-sampling tell that the output was generated rather than observed.

In all four cases the user noticed, 16–39 s later. The agent never self-detected, and there is no harness-side check.

Mechanism

assistant/compaction.rs:

if matches!(strategy, CompactionStrategy::SessionRotationSummary) {
    return Ok(fallback_summary(messages));   // never calls a model
}

fallback_summary renders the window through render_content_parts, which serialises history as a text transcript including both halves of every tool interaction:

ContentPart::ToolUse   { .. } => format!("[tool call: {} {}]", tool_name, truncate_json(..)),
ContentPart::ToolResult { .. } => format!("[tool result: {}]", truncate_json(..)),

That digest is stored as a MessageRole::System message, and providers/anthropic.rs::build_request_body folds every System-role message into the request's system block.

So the model begins its turn having just read up to ~800 KB of interleaved call/result transcript in system position, with no marker for where the transcript ends. It continues the document instead of replying: it writes a call in prose, and because prose contains no protocol boundary that returns control to the harness, generation does not stop — so it writes a result too, reconstructed from the stale transcript it just read.

Two explanations that the data rules out

• Not format imitation. 3 of the 4 incidents used <invoke name="…"> / <parameter name="…">, a syntax that appears nowhere in the digest — it is the model's own text-mode prior. Only one echoed the digest's [tool call: shape. Changing the rendering format therefore would not fix this.
• Not conflicting instructions. No text-mode function-calling preamble is injected anywhere; grep -rn "invoke name\|function_calls" src-tauri/src src is empty. The native tools parameter is the only affordance offered.

What correlates is the size and position of the digest, not its wording.

Secondary defect: rotation compaction does not compact

Each rotation digest re-embeds its predecessor verbatim, so summaries grow monotonically — 450 KB → 796 KB across 17 nestings, roughly +45 KB per cycle. Nesting depth was measured by counting occurrences of the summary preamble inside a single summary message. A ~200K-token "summary" in the system block inverts the purpose of the feature, and it is why incidence rises over a long-lived session.

Suggested fixes, in the order the evidence supports

1. Harness-side detection. Before persisting an assistant message, scan its text parts for an invocation shape (<invoke name=, <parameter name=, [tool_call:, [tool call:). On a hit, do not store it as a normal reply — discard and re-prompt with an explicit correction, or fail the run visibly — and log the hit so the rate is measurable. This is the only proposal that catches the fabricated-result case, and the only one that does not depend on getting prompt wording right.
2. Do not deliver a transcript in the system block. Send the digest as a user-role turn terminated by an explicit end-of-history boundary. Restores turn structure so the next token is a reply rather than a continuation, and removes instruction authority the transcript has not earned.
3. Bound the digest and break the nesting. Never copy a prior compaction summary forward verbatim; re-summarise or truncate.
4. Use the summariser model on rotation when a provider is reachable, keeping fallback_summary for genuine failures. Model-summarised digests showed zero incidents.
5. Change the [tool call: rendering — hygiene only; demote, since it explains at most 1 of the 4 cases.

Items 1–3 are independent and can land separately. Item 1 converts a silent data-integrity failure into a visible error and is worth shipping on its own.

Detection query

Parse content_json per content part rather than pattern-matching the raw JSON — matching raw text yields false positives from tool-call arguments (a query searching for these markers matches its own SQL):

import sqlite3, json, re
con = sqlite3.connect("file:<workspace>/.clai/data.sqlite?mode=ro", uri=True)
pat = re.compile(r'<invoke name=|<parameter name=|\[tool_call:|\[tool call:')
for mid, cj in con.execute("SELECT id, content_json FROM assistant_messages WHERE role='\"assistant\"'"):
    parts = json.loads(cj)
    if any(p.get("type") == "text" and pat.search(p.get("text", "")) for p in parts):
        print(mid, [p.get("type") for p in parts])

Messages with a text-part hit and no tool_use part are the failures.

Environment

• CLAI v26.8.1 (b5a9564)
• Provider: Anthropic protocol adapter
• Linux, webkit2gtk

[juacker]

Proposed fix: reference-based digest instead of an inlined transcript

Rather than serialising tool calls and results into the digest text, emit a one-line record per tool call with its tool_call_id, and let the agent pull the full payload on demand via history_query. The payload is already durably stored in assistant_tool_calls.result_json; the digest only needs to point at it.

This attacks both properties that correlate with the failure — the digest's size and its shape as an unterminated transcript — rather than just its wording.

Why the ids are already available

compaction.rs::render_content_parts currently throws them away:

ContentPart::ToolUse { tool_name, arguments, .. } => Some(format!(
    "[tool call: {} {}]", tool_name, truncate_json(arguments, tool_call_max)
)),
ContentPart::ToolResult { payload, .. } => Some(format!(
    "[tool result: {}]", truncate_json(payload, tool_result_max)
)),

Both .. patterns discard the id (ToolUse.id, ToolResult.tool_call_id). Rendering a reference is a change to this one function plus the recovery preamble in fallback_summary.

Retrieval side needs nothing new:

SELECT tool_name, params_json, result_json
FROM assistant_tool_calls WHERE id = 'toolu_…';

The stub line is the actual design work

A reference only avoids a re-fetch if the agent can tell from the stub whether the content matters. bash_exec ok 312B is useless. Something closer to:

bash_exec · cargo test -p assistant · ok · 2 failures in compaction.rs · toolu_01ABC…
fs_read   · src/compaction.rs · ok · 612 lines · toolu_01DEF…

carries enough signal to skip. Get this wrong and you trade fabrication for a quieter failure: the agent proceeds on thin context and never knows what it missed.

Suggested hedge: keep a short inline excerpt for the most recent N calls (where the agent is most likely to need detail) and drop to pure references for everything older. Recency is also where the current digest is cheapest to keep.

Render references in a non-invocation shape

No brackets, no JSON, no name(args). A flat record line, as above. Nearly free, and it removes the last surface that can be imitated.

Note this alone would not have prevented 3 of the 4 observed incidents — those used <invoke name=…>, which appears nowhere in the digest. The size/structure reduction is what matters; the shape change is belt-and-braces.

Worth pairing: move the digest out of the system block

The digest is written as MessageRole::System (compaction.rs ~193 and ~352) and the Anthropic adapter folds System messages into the system block. The model therefore opens its turn having just read an unterminated transcript in maximum-authority position, and continues the document instead of replying to the user. Shrinking the digest lowers the odds; delivering it as a user-role message removes the posture that produces the behaviour. The two changes are independent and complementary.

This does not fix the nesting growth

Each rotation digest re-embeds the previous one verbatim (450 KB → 796 KB over 17 nestings, ~+45 KB/cycle). References shrink each layer, but the recursion still compounds. Flatten prior digests instead of nesting them, or the small digest grows back into a large one by rotation ~30.

Open questions

1. Tool availability. Checked — not an issue. history_query is registered under
   if context.agent_workspace_id.is_some() (tools/registry.rs), but no production path
   produces None: resolve_workspace_descriptor always sets root_path: Some(..)
   (commands/workspace.rs:919), and agents/runner.rs:541 and
   tools/workspace_tasks.rs:492 both hardcode Some(..). The only root_path: None in the
   tree is a test helper. Treat "the digest's retrieval tool is always available" as an
   invariant to preserve rather than a gap to design around.
2. Dangling references. Grep found no retention, pruning, or VACUUM path over assistant_tool_calls — the only deletes are single-message ones on the edit/queue path, which do not touch the tool-call rows. So references look durable today, but this becomes an invariant that any future retention feature must respect.
3. Retrieval cost. Worst case the agent re-fetches the same large payload across several turns, and the saved tokens come back. The stub quality in the section above is the main lever; a note in the recovery preamble ("prefer the stub; fetch only when the detail changes your next action") is the cheap complement.

Side benefit: fabrication becomes checkable

With ids in the digest, a claimed tool result can be validated against the record. A result asserted in assistant text with no corresponding assistant_tool_calls row for that run is provably invented — which is the detection query this issue currently lacks, obtained for free.