feat: record token usage, cost and latency per agent and per run - #78
Merged
Merged
Conversation
Every provider reports token usage and the extension discarded it. This reads it into a Usage record on ChatResponse, normalised across providers after the OpenTelemetry GenAI conventions: input counts every prompt token including cached ones, output counts every generated token including reasoning, and provider-specific extras (reasoning, cache read/write, OpenRouter's reported cost) are kept where reported. The HTTP layer stamps wall-clock latency on each call, retries and throttle waits included. Two reporters expose it in the same [key value] shape as structured output: llm:usage for the calling agent and llm:usage-total for the run. Both reset on clear-all. A reply the extension then rejects still counts, since the provider billed it; a call with no reply records nothing. Tests: UsageParsingSpec covers each provider's field mapping and the latency stamp; eight tests.txt cases cover accumulation, per-agent isolation, async, every primitive, rejected replies and failed calls via a __TEST_USAGE marker in the test provider. T9 in the live harness verifies against Groq: 34/34 pass. Closes #49
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #49.
What
Every provider returns token usage and the extension threw it away. This PR reads it, normalises it across providers, and exposes it to the modeler.
Usagerecord onChatResponse: input, output, total, plus optional reasoning, cache-read, cache-write, cost, latency.costis passed through only when a provider reports it (OpenRouter). Nothing is estimated and there is no price table to go stale.latency-msis stamped inBaseHttpProviderfor every call, throttle queueing and rate-limit retries included.[key value]shape as structured output sollm:getreads them:llm:usage(calling agent) andllm:usage-total(whole run). Both reset onclear-all.llm:choose) still counts: the provider billed it. A call that never produced a reply records nothing.Tests
UsageParsingSpec: 13 tests, one per provider field mapping plus absent-usage cases,Usage.plus, and two latency tests (stamped without usage; covers a 429 backoff).tests.txt: 8 cases via a__TEST_USAGE:in,out[,reasoning]marker inDeterministicTestProvider: starts at zero, accumulates, per-agent isolation, resets onclear-all, async, every primitive, rejected reply counted, failed call not counted.demos/e2e-tests. Against Groqopenai/gpt-oss-20b, 34/34 pass. Groq reports reasoning tokens for this model and the parser picks them up.Deterministic suite: 224 passing.
Docs
docs/API-REFERENCE.md: new Token Usage section with the key table, the per-provider mapping, and thecostis""when unknown rule.