The chat loop runs on rig

[WaylandYang]

Spike for #546. Draft: this is the branch the issue asked for, run against the harness, with the report below. Nothing here is meant to merge until the findings are read.

What changed

The hand-written loop in chat.rs is gone. chat.rs now assembles a rig agent, drives its stream, and translates rig's items into the SSE frames the frontend already consumes. Three new pieces:

• api/rig_model.rs: LlmClient behind rig's CompletionModel. rig's request (preamble, history, documents, tool definitions, tool_choice) is spelled out as the OpenAI-protocol JSON LlmClient already speaks; LlmClient's turns and streams come back as rig's responses. The wire stays ours: read timeout, error bodies (LLM errors preserve their response bodies #538), out-of-credit versus rate-limit classification all survive, and the whole anyhow chain travels through rig's CompletionError so the loop can still classify a failure. Two decisions live here: rig's "documents" (the entities identified earlier in the conversation) become a system message placed right before the current question, not a fake user message (The chat loop is hand-written, and every policy lands as another branch #546's second finding: the model used to answer that block as if the user had written it); and ToolChoice::None sends no tools field at all, which every endpoint accepts, instead of "tool_choice": "none", which not every endpoint does.
• api/agent.rs: the tools and the policy. The eight tools are rig DynamicTools built from tools_schema (the list MCP also uses), still executed by tools::dispatch; a ninth, no_evidence_needed, is the terminal for questions that are not about the knowledge base. The policy is one AgentHook: on_completion_call sets tool_choice: required until some tool has run and, on the request after the budget, withdraws the tools and orders an answer; on_tool_call runs check_call (unchanged, its tests unchanged) and skips a call that fails it, feeding the model the same message as before; on_tool_result hands the tool's UI step to the stream; on_model_turn_finished catches an endpoint that ignores required (text-only first turn, no tool run) and sends it back once.
• utopia-llm: a typed Rejected error and a tool_choice pass-through. Non-2xx responses that are neither out-of-credit nor rate-limit now carry their status code as a type (the text is unchanged). The loop degrades to one-shot RAG only when the first request with tools comes back 400/422, which is what "the endpoint rejects tool calling" looks like on the wire. Before, any first-round error did that (The chat loop is hand-written, and every policy lands as another branch #546's first finding).

The #543 guard (STALL_NUDGE, DONE, the no-evidence note) is deleted: termination is structural now. A turn cannot end before a tool has run, so "请稍等，我将调用工具" is no longer a possible final state.

Unchanged: the SSE contract (conversation, step with the UTF-16 at, sources, delta, done, error, same order), reconnect through live, the four persisted fields (steps, sources, resolved, tool_exchange; the next turn replays them, now converted to rig messages), the record axis (tools untouched), the budget answer, the tool list MCP sees.

The harness

The fifteen turns from #543's testing, DeepSeek-V3 through SiliconFlow, same scratch database. All fifteen finished with done; no error, no degradation, no stall.

turn	question	tools run	sources	verdict
1	openai和anthropic关系	no_evidence_needed	0	answered from memory; reason given: "general knowledge outside the scope of the user's data"
2	OpenAI 的时间线	find_entities	0	asked which of the several "OpenAI" entities
3	说短一点	no_evidence_needed	0	correct terminal
4	你好	no_evidence_needed	0	correct terminal
5	Sam Altman 被罢免时董事会有谁	find_entities, entity_facts	0	board listed from the graph
6	openai和anthropic的关系	no_evidence_needed	0	answered from memory; "external companies not in the user's knowledge base"
7	OpenAI 的时间线	find_entities, entity_facts	0	full timeline from 433 facts
8	你搜了吗	no_evidence_needed	0	"yes, from the knowledge base": true this time (turn 7 did)
9	啥情况	no_evidence_needed	0	honest follow-up
10	你是谁	no_evidence_needed	0	correct terminal
11	openai和anthropic关系	search_chunks	6	cited answer
12	你是谁	no_evidence_needed	0	correct terminal
13	openai和anthropic关系	search_chunks	6	cited answer
14	OpenAI 的时间线	no_evidence_needed	0	"not in the knowledge base", without looking; reason: "OpenAI is not part of the user's data"
15	你搜了吗	find_entities	0	listed the OpenAI entities

What fit

• rig's runner carries the loop with no branch of ours left in it. The policy is 120 lines of hooks; chat.rs lost 630 lines.
• SiliconFlow honours tool_choice: required: every first turn called a tool. The fallback for an endpoint that ignores it (the turn-finished hook) never fired; the fallback for an endpoint that rejects it (RigModel resends without tool_choice) never fired either. Both are in place, neither has been exercised by a real endpoint yet.
• The stall is gone. Under A promise to search is not an answer #543's loop the guard fired four times in fifteen turns and the model answered DONE every time, including after promising a search. Here no turn ended on a promise, and no hidden second request was made.
• The stream and the stored turn match the old shapes exactly (frame order checked on turns 7 and 11; the four fields on every stored turn).
• The two loop defects from The chat loop is hand-written, and every policy lands as another branch #546 are closed by construction: a network failure on the first request is an error frame, not a degradation; the entity block is a system message.

What did not

Structural termination guarantees a decision, not a correct one. Three of the six fresh data questions (turns 1, 6, 14) ended with no_evidence_needed, and the stored reason is the same each time: the model assumed the knowledge base does not hold OpenAI. It had find_entities one call away and did not spend it. Turn 14 is the worst shape, a claim about the base's contents made without looking. The same question in turns 11 and 13 was searched and cited.

What is different from before is that the miss is now an explicit, logged, stored decision: the call is in tool_exchange with the model's reason, and sources is empty. That makes #547's "no sources" mark exact, and it gives a number to watch per model. Under #543 the same miss was an absence.

Not tried, on purpose: rewording no_evidence_needed's description or result, or filtering the question's text. Both are the "more prompt wording" the issue ruled out. Two structural candidates, neither measured:

• withhold no_evidence_needed from the first request of a conversation whose message names something (an entity lookup first, then the terminal is available) — this is text matching by another name and I do not recommend it;
• make the terminal's result carry a fact the loop knows and the model tends to forget: that the base was not consulted, so nothing may be claimed about its contents. That is one sentence in a tool result, not a prompt, and the harness can tell whether it moves the three misses. It is the cheapest next measurement if the number matters.

Dependencies

rig-core and rig-agent 0.42, both with default-features = false (no provider clients, no TLS stack of their own). Seventeen small crates, schemars the largest.

Tests

• rig_model: preamble first, documents as a system message before the question, tool exchange round-trips in protocol shape, tool_choice mapping, None drops the tools field, cut-off arguments stay a string (so check_call still refuses them), a missing call id is minted.
• agent: the last exchange is replayed before its conclusion with tool names recovered; the entity block's shape and limit.
• check_call's six tests unchanged; utopia-llm's seventeen unchanged (error text identical).

To decide

Whether the loop is replaced (this branch, plus the decision record) with the three misses accepted as the model's and #547 as the surface for them, or whether the terminal-result measurement above runs first.

🤖 Generated with Claude Code

[WaylandYang]

Two more runs to separate the model from the prompt, both on this branch, same scratch base, twelve fresh data questions each (relation ×3, relation-again ×3, timeline ×3, board ×3).

1. Prompt redefined, same model (DeepSeek-V3). The reasons the model gave for no_evidence_needed quoted the prompt: "outside the scope of the user's data". The prompt framed the tools as being for "the user's data" and never said what the base holds. Commit 3 replaces that phrase and adds five lines: the base holds whatever was ingested, you do not know what is in it until you look, "general knowledge" is never a reason to skip the tools, never say the base lacks something you have not searched for.

	fresh data questions	skipped the base	reasons given
before	6	3	"not in the user's data" ×3
after	12	3	"general knowledge" ×2, "ambiguous, cannot answer without clarification" ×1

Halved, not gone. Two of the three remaining skips cite "general knowledge" while the prompt now says in those words that general knowledge is never a reason. The model reads the instruction and overrides it.

2. Same prompt, other model (Qwen2.5-72B-Instruct on the same endpoint; Qwen3-235B is disabled on this key). 12/12 called a real tool first; no_evidence_needed was never called. Qwen has its own habits (it narrates its plan, and it asks which "OpenAI" more often than it picks one), but it does not answer a data question from memory.

So: the skips are DeepSeek-V3's; the prompt's "user's data" framing made them more likely and is fixed on this branch. Every skip that remains is a stored no_evidence_needed call with the model's reason, so #547 can mark it and the rate can be read per model.