fix(dashboard): stop re-pricing finished history from live counters - #142
Merged
Merged
Conversation
A completed day changed value whenever the process restarted. The same 30-day window read $32.58, $85.88 and $16.83 within an hour today, and the day of 2026-08-30 — a fixed 12,604,819 tokens in every reading — was reported at $7.83, $21.74 and $4.07. Only the multiplier moved. Buckets recorded before the per-model column existed carry token counts but no split, so they can only be valued at an average rate. That average came from blendedRate(metrics, ...), which reads the in-memory model counters — counters that reset on restart and reflect only what has served since. Nine days of history, 94% of the window, were priced by whichever models happened to have traffic in the current process. The blend now comes from the per-model splits stored in the window itself, so it is a function of persisted data and prices the same window the same way twice running. It is still an estimate and the card still says so; it is now a reproducible one. The live mix remains the fallback for a window that carries no split at all. Model colours had the same root cause: they were assigned from the lifetime per-model earnings, also in-memory, so after a restart every model that had not yet served folded into "Other" despite the history holding its figures. The stored history is merged in, ranking each model on whichever total is larger. On this node that took the 30-day breakdown from one named model to four.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A finished day should not change value
The same 30-day window, read three times in one hour on the live node:
Window totals: $32.58, $85.88, $16.83. The token counts are identical in all
three readings — the history did not change, only the multiplier.
Why
Buckets recorded before the per-model column existed carry token counts but no
split, so they can only be valued at an average rate. That average came from
blendedRate(metrics, rates), which readsmetrics.ModelMetrics— thein-memory counters, which reset on restart and hold only what has served since.
Nine days of history, 94% of this window, were priced by whichever models
happened to have traffic in the current process. Three restarts today, three
different valuations of the same past.
The blend now derives from the per-model splits stored in the window, making
it a function of persisted data. Still an estimate — the card says so — but a
reproducible one. The live mix stays as the fallback for a window carrying no
split at all.
The same bug wearing a different hat
Model colours were assigned from lifetime per-model earnings, also in-memory.
Straight after a restart that list held one model, so the other five folded
into "Other" even though the history had full figures for them:
Merging the stored history in, ranking each model on whichever total is larger:
Test
TestHistoryPricingDoesNotDependOnLiveCountersprices one window against twovery different live mixes. Without the change: $6 vs $300.
Not addressed here
The large
Unattributedfigure is correct, not a defect — those nine dayspredate the per-model column and genuinely cannot be attributed. Worth a
separate look at how the panel communicates that, since it dominates the window.