feat(usage): record what a request cost, per session and per weekly family - #192
Open
ak2k wants to merge 3 commits into
Open
feat(usage): record what a request cost, per session and per weekly family#192ak2k wants to merge 3 commits into
ak2k wants to merge 3 commits into
Conversation
A session held one pin, so a routing decision taken for one model family relocated the whole session. An account whose Fable weekly is spent still serves Opus, so a Fable request diverts to another account, re-pins the session there, and the next Opus request follows it onto an account nothing evaluated for Opus and whose Opus cache is cold. Whichever family routed last owned the session. Key each pin by the weekly bucket that governs the request, using the key selection already applies for eligibility, so the families keep independent affinities. recordSession takes the request's model to name the bucket it spent; session selection looks up the executor's. The load metric follows the pin: a session counts on every account it is currently spending, once each, and a pin stops counting once its bucket has been quiet for the active window. Finishing a request ages the pin it was spending, so an account does not drop out of the metric the moment it completes a long stream. removeAccount renumbers session pins the way it already renumbers route pins.
…ring quota first Rotation only chooses an account at the moments it is forced to: daemon start, and an account crossing the switch threshold. On a fleet whose weekly utilization never reaches that threshold those moments never arrive, so routing stays on whatever account it last landed on — often the one whose window just reset a full week out — while another account's ample weekly quota quietly expires unspent. The existing soonest-reset preference is not wrong; it simply never gets consulted. Score each account by expiry pressure — headroom in the weekly bucket that governs the request, over the seconds until that bucket resets — and draw selection from the top band: the accounts within a tolerance ratio of the best pressure in the top priority tier. Headroom is the numerator, so a nearly drained account is not preferred merely because its window rolls soon. Both halves of the ratio come from one bucket, so a family utilization is never divided by the shared window's clock. A band rather than a sort keeps the comparison transitive and leaves distributeSessions free to spread load across near-equal accounts; banding only the best priority tier keeps an operator's explicit order winning outright. Re-route a sticky choice — a session pin, or the current account — on one event only: its governing weekly window rolling over, the moment that account becomes both the freshest and the furthest-dated. Draining it never preempts, since the drain is that session's own traffic and moving on it would spend a prompt-cache miss per crossing while the same account is still the one worth spending. Off by default. With the knob off, every routing decision is byte-identical to the previous behaviour; the status payload gains an expiryRouting echo and a per-account pressure figure either way, since pressure is a measurement of the fleet rather than a report of the feature's state.
…amily RELEASE NOTE. `teamclaude status --json` gains token accounting: `sessions.tokens` (cache reads, cache creation, input, output, the report count backing them, and the live cached footprint), the same figures split per weekly bucket under `sessions.tokens.byBucket`, and per-account `usage.totalCacheReadTokens`, `usage.totalCacheCreationTokens` and `usage.byBucket`. Routing is unchanged. Upstream reports what a request cost in a `usage` object. Two of its four fields were read and the rest discarded: `updateUsage` took `input_tokens` and `output_tokens`, and nothing looked at `cache_read_input_tokens` or `cache_creation_input_tokens` at all. `input_tokens` counts uncached input only, and a Claude Code turn reads a large cached prefix while sending almost no fresh input, so the field being counted measured close to nothing. Over 873012 usage objects carrying an input side, from 1540 sessions in local transcripts, the medians are input_tokens 2 and cache_read_input_tokens 199824; summed, the counted field carries 0.05% of the input side and the two discarded ones carry the other 99.95%. Nothing was attributed to the session that caused the spend either. Totals are kept PER WEEKLY BUCKET rather than once per session or once per account. Fable meters into its own weekly window, so what a point there costs is its own question, and a sum across families cannot be taken apart again afterwards: the distinction is gone before anything can use it. The bucket comes from `_weeklyBucketFor`, the same resolver routing uses, so a total and a routing decision never disagree about which family a request belonged to, and a route's `bucket` override moves both together. THE STREAMING PATH RECORDS ONCE PER MESSAGE. A streaming response reports its usage twice, and the second report is CUMULATIVE for the whole message rather than an increment on the first: `message_start` carries the input side with a placeholder output figure, and `message_delta` supersedes every field it carries. Handing each event's object straight to the recorder therefore double counts whatever appears in both. The events are merged into one object as they arrive and written once in `streamResponse`'s finally, which makes the double count unrepresentable rather than merely avoided, and records the input side of a stream that died after `message_start`, since those tokens were spent upstream whichever way the stream ended. An empty merge is written nowhere rather than written as zeroes: a ping-and-text-deltas stream carries no usage, and recording one would report an observation that never happened. `updateUsage` keeps its existing incremental, per-event behaviour and its existing counters. It is on the path every current caller and test drives, and the new call adds a second scope whose lifetime is not the account's, so the two stay apart. `context` is the one figure that is not a sum. It is the size of the last context upstream reported reading, which is what one request on this session costs to serve; summing it across a conversation would answer a question nobody asks. A report carrying output alone says nothing about context size and leaves it standing rather than resetting it to zero. `reports` counts the usage objects that contributed, so a reader can separate "no tokens because the session is idle" from "no tokens because nothing was ever observed". Those two states look identical at zero, and the second of them means the pipeline is broken. The per-session totals live on the session record beside its pin, so there is one lifetime and one sweep for everything scoped to a session. A report for a session the tracker has already forgotten is dropped rather than creating a record: the id is a client-supplied header, and creating records on the usage path would let responses repopulate a map the idle window exists to drain. NOTHING ROUTES ON ANY OF THIS. Every figure is published and none is read back by selection. Both trees were driven over identical seeded fleets and schedules with the recording live on the branch throughout: 200000 picks per `distributeSessions` arm, 0 differing, 349050 usage reports recorded across the two arms. Tests: test/session-token-accounting.test.js (22), test/streaming-usage-merge.test.js (7) and test/usage-attribution.test.js (7). The merge file drives the real proxy rather than the tracker, because the double-count defect lives at the call site and no tracker-level fixture can reach it. One of its cases exists specifically for the `finally`: an upstream that writes message_start and then destroys the socket, which is the only shape where the record is written from a rethrowing path. Every other streaming case ends cleanly, so moving the record onto the success path left the whole suite green until that case existed. REBASED ONTO THE TWO ROUTING CHANGES THAT PRECEDE IT IN THIS SERIES. This change was written against master and is replayed here onto the per-bucket session pins change and the expiry-pressure routing change. Both it and the pins change restructure `src/session-tracker.js`, so the replay conflicts there and nowhere else: `src/account-manager.js` and `src/server.js` merge clean. Four hunks, each resolved with the reason, because the resolution is the one part of a rebase that is a decision rather than a replay: 1. The record-shape docblock on the constructor. Union. The two sides describe one map from opposite ends, `pins` and `windows` from the pins change and `tokens` from this one. Neither is wrong, so the comment names all of them. 2. `_ensure`'s record literal. The pins side wins on identity; this side adds `tokens`. `accountIndex` is deliberately not carried forward: the pins change replaced that single scalar with a pin per weekly bucket, and reinstating it would leave two answers to "which account is this session on", which is the question that change exists to stop anyone asking. `tokens` keys on the same bucket space as `pins`, so a family's spend and the account it is pinned to are one lookup apart. This is the only hunk where the resolution is a judgement, and it is the one to read first. 3. `_live` and `pinnedAccount`. The pins side, outright. This change introduced a `_live` helper on a base that had none; the pins change already carries an equivalent with a defaulted clock, and all three call sites here pass their clock explicitly, so the two are interchangeable at every one of them. Keeping both would define `_live` twice in one class, where the second definition silently wins and the first reads as live code. The single-pin `pinnedAccount(sessionId, now)` goes; the bucket-keyed `pinnedAccount(sessionId, bucket, now)` replaces it. 4. `stats`' active branch. Union, both loops kept. They compute different things over the same record: the pins change's walk fills `perAccount` once per account a session is currently spending, and this change's walk accumulates the live cached footprint per bucket. Dropping either loses a published field. No behaviour of this change was altered to make it fit, and nothing in the pins or routing changes was altered at all. Every line of the resolved file is one side's or the other's, except for two comment lines explaining hunk 2, the docblock line in hunk 1 that names both sides' fields, and one object literal reflowed onto several lines so `tokens` could be added to it. Suite on the rebased tree (`node --test`): 659 tests, 659 pass, 0 fail, 0 cancelled, 0 skipped. The two changes it sits on carry 623 tests between them and this one adds 36, so 659 is every test from both sides and the resolution dropped none. Routing is unchanged by the rebase as it was by the change: both trees driven over identical seeded fleets and request schedules, with session distribution and expiry routing each on and off, 800000 picks compared, 0 differing, 700241 usage reports recorded. A control that lets the recording path nudge a quota level produces 9730 differing picks under the same drive, so the zero is a measurement. The differential is evidence about routing alone; the suite covers the published stats fields and the session lifetime behaviour.
ak2k
force-pushed
the
upstream/session-token-accounting
branch
from
August 25, 2026 01:05
f16b5f0 to
f4d0352
Compare
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #189 and #190; review those first, and note the diff here spans all three commits, only the last of which is this change. Merging out of order will conflict. Rebased 2026-08-24 so the accounting sits on the session-pin structure it measures. It inherits #190's collision with #191 (one import name and one adjacent export, in
src/account-manager.jsandsrc/model.js); if #191 lands first, #190 and this rebase together, the same day.The proxy reads one field out of every usage object and discards the rest. Measured over 873,012 usage objects from 1,540 local sessions: the counted field (
input_tokens, median 2) carries 0.05% of the input side; the two discarded fields (cache_read_input_tokens, median 199,824, andcache_creation_input_tokens) carry 99.95%.What this adds
Capture
input_tokens,cache_read_input_tokens, andcache_creation_input_tokensper session, per weekly family bucket, and publish them in the status payload (sessions.tokens, per-bucket and per-account splits). Instrumentation only: nothing here reads the numbers back into routing, and no selection path changes. It gives an operator visibility into what sessions actually cost, and any future load-aware feature a measured foundation instead of a guess.The one subtle part
Streaming responses report usage as a
message_startplus cumulativemessage_deltas. The deltas are merged and recorded once, in afinally, never re-added per event: a stream reportinginput_tokens12 at start and a cumulative 10,682 in its delta records 10,682, not 10,694. A stream aborted mid-flight still records the input it already spent, because only thefinallyruns. Both properties have direct tests driving the real proxy over a socket.Routing unchanged, measured
Both
distributeSessionsarms, seeded mixed-model schedules (four families, 90 to 108 requests per family per run), recording live on this branch throughout: 400,000 picks compared, zero differing, 349,050 usage reports recorded while comparing. A negative control (a branch copy whose recorder also nudges quota by 0.001) trips the same probe 1,896 and 3,203 times on the two arms, so the zero is a measurement, not an artifact.Tests
525 before, 561 after, 0 fail measured against master; on the rebased branch, 623 before and 659 after.
npx eslint .clean. No existing test file is touched. 18 targeted mutations (field discarded, count frozen, wrong session charged, delta re-added per event, merge guard removed, published split aliased, record moved out of thefinally, and others), each reverted individually: every one turns at least one named test red. The full reversion with new tests parked passes upstream's suite 525/525, which measures that no existing test covers this path.Compatibility
Additive fields only; nothing existing changes shape. Counters are since-process-start (
restoreQuotaStaterestores quota, not usage). The per-session and per-account splits are asymmetric by design (six figures per family on the session side, two on the account side); the payload documents which.Limitations