feat(engine): record agent spend on every phase exit (U1) - #25
Merged
Merged
Conversation
agent_end was emitted only when an agent phase entry succeeded, so any cost figure undercounted every failed, dead, or attempt-ending entry. Every entry that emits agent_start now closes it with agent_end carrying an outcome (passed, failed, died, aborted, cancelled, ceiling, send_budget) and an unmetered_sends count, before the event that closes the phase or the attempt. A runtime-error send's reported usage is now added to spend. A send killed in flight (watchdog, phase clock, ceiling, cancel) or ending without a usable result is counted as unmetered, not as free. Claude Code's total_cost_usd is the session's running total across --resume, not the invocation's cost: a real haiku resume reported 0.0585011 after a first send of 0.0405199. The adapter now records the session's reported total on runtime.Session and returns each send's own share, so the engine's per-send sum is correct. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A discontinuity is a send death and the engine drops that session, so the recorded total is never read again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l entries Mutation testing showed the abort paths inside fail and terminalExit could drop agent_end without a test noticing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedNext included review available in 46 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements U1 of the factory-quality follow-ups plan (PR #21): R1, KTD2.
What changed
agent_endon every agent phase exit. Before this, it was emitted only when an entry succeeded, so every failed, dead, or attempt-ending entry dropped out of any spend figure. Now every entry that emitsagent_startcloses it with exactly oneagent_end. The event comes before whatever closes the phase or the attempt (phase_end,phase_death,attempt_send_budget_exhausted,attempt_ceiling_exceeded). The payload adds:outcome, one ofpassed | failed | died | aborted | cancelled | ceiling | send_budget. These are pinned asprotocol.Agent*constants for the report (U2).unmetered_sendssendreturns (KTD2).sends = metered + unmeteredper entry.total_cost_usdturned out to be cumulative (see below). The adapter records the session's reported total on a newruntime.Session.ReportedCostUSDand returns each send's own share asUsage.CostUSD. The engine's+=is unchanged and is now correct. Sessions are only ever built fresh in memory (deaths and CI-repair rounds start new ones), so the running total never resets mid-conversation.total_cost_usdsemantics: cumulative across--resumeTwo real calls,
claude2.1.283,--model haiku,-p "say ok":usage(per invocation)total_cost_usdmodelUsagetokensusage--resumesame id)Pricing call 2's own
usageat Haiku 4.5 rates gives $0.0179812, and 0.0405199 + 0.0179812 = 0.0585011 exactly. Sototal_cost_usdis the session total, whileusageis per invocation. Before this fix, every resumed correction send was double-counting all earlier sends in its session.Verification
just checkpasses (format, vet, boundary, definitions, tests, build).go test -race ./internal/engine/... ./internal/runtime/...passes.status: fail, runtime error, death with re-entry, send-budget mid-phase, send budget refusing an entry's first send, ceiling, cancel, and boundary abort;agent_end.costequals the usage of the sends that returned a result, and summedunmetered_sendsequals the killed and crashed sends.agent_endoutcome. A new exact-sequence test pins the new exits in order: crash →agent_end(died)→phase_death→ re-entry → pass → next phase refused by the send budget →agent_end(send_budget)→attempt_send_budget_exhausted.Mutation testing
18 mutants were tried, and each one compiled (
go vet), so no compile error was counted as a catch. The first run killed 16. The two survivors were theagent_endcalls on the breach (abort) branches inside the failed-entry and attempt-terminal exits. I addedagent_endassertions to the two existing boundary tests that drive those branches, and both mutants are now killed: 18/18.The mutants also covered: runtime-error metering, the unmetered count on kill and on crash, the
agent_end→phase_deathorder, every outcome label, the payload field, and both adapter delta lines.Known gaps / untested
ReportsCost=false), so it's unaffected.failInfraexits (snapshot, seed, prompt read) happen beforeagent_start, so they correctly emit noagent_end.🤖 Generated with Claude Code