spec: make budget accounting concurrency-safe - #46
Open
santosomar wants to merge 2 commits into
Open
santosomar wants to merge 2 commits into
santosomar wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
A few newly-updated docs contain internal terminology/contract inconsistencies (e.g., “maximum output allowance” vs “maximum output-token allowance”, “cached input” vs “cached-input”, and Sync Impact Report regeneration wording) that should be reconciled before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR bumps the spec to v0.2.0 and strengthens the budget-governor requirements to be concurrency-safe and deterministic by specifying atomic cost reservation + exactly-once settlement, active-runtime semantics, deterministic trailing yield windows, and idempotent stop/hot-reload behavior.
Changes:
- Define atomic spend admission via conservative maximum-cost reservations (FR-112a) and an idempotent, attributed per-call accounting ledger (FR-113).
- Make yield computation deterministic via a precise trailing spend-window suffix definition (FR-115) and add an idempotent stop transition (FR-116a).
- Update downstream reference, adoption, operations, governance, glossary, and changelog docs to reflect the strengthened budget semantics and spec version bump.
File summaries
| File | Description |
|---|---|
| spec.md | Bumps spec to 0.2.0; adds/strengthens FR-011, FR-112/112a/113, FR-115/116/116a, FR-128, plus new SC-010 and related terminology updates. |
| README.md | Updates status badge to Seed v0.2.0 and trims trailing blank line. |
| GLOSSARY.md | Aligns glossary definitions for Yield and Budget governor with the new settled-spend / reservation semantics. |
| docs/reference/terminology-quick-card.md | Updates quick-card terminology for Budget governor and Yield to match the new definitions. |
| docs/reference/principle-fr-matrix.md | Updates principle-to-FR mapping to include FR-112a and FR-116a under relevant principles. |
| docs/reference/open-questions-checklist.md | Expands the §11.2 checklist item to include billable token categories, direct cost, and rate-card source. |
| docs/reference/fr-index.md | Updates FR summaries and adds index entries for FR-112a and FR-116a. |
| docs/operations/observability-checklist.md | Expands budget/yield observability requirements (settled vs reserved spend, admission state, window fullness, rate-card identity). |
| docs/operations/budget-and-stop-conditions.md | Adds concrete admission/settlement pseudocode, active-runtime semantics, hot-reload/stop behavior, and conformance cases. |
| docs/governance/sync-impact-reports.md | Archives the spec 0.1.0→0.2.0 Sync Impact Report and updates wording around when new entries are added. |
| docs/architecture/substrate-contracts.md | Introduces an explicit “budget governance contract” API surface and required guarantees/anti-patterns. |
| docs/adoption/integration-decisions.md | Updates required integration capabilities for maximum-output-token allowance, token categories, and immutable pricing snapshots. |
| docs/adoption/clarification-playbook.md | Updates the §11.2 provider clarification guidance to include max-output allowance and rate-card/versioning considerations. |
| constitution.md | Updates the Sync Impact Report header block to reflect the spec 0.1.0→0.2.0 change (constitution unchanged). |
| CHANGELOG.md | Adds changelog entry for spec v0.2.0 and related documentation updates. |
Review details
- Files reviewed: 15/15 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | **A-1** | The operator is authorized to evaluate the target and to probe the testbed; authorization is established outside the system. | Out of scope (§14). | | ||
| | **A-2** | The operator has read access to the target's source code. | §1.5 clarification; black-box mode is out of scope (§14). | | ||
| | **A-3** | A frontier LLM with tool/function calling is available via API, with per-call token accounting. | §11.2 clarification. | | ||
| | **A-3** | A frontier LLM with tool/function calling is available via API, with a caller-set maximum output allowance and per-call accounting for billable token categories. | §11.2 clarification. | |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The deterministic trailing-window requirements rely on a durable settlement ordering key, but the accounting-event field list does not currently require storing the settlement timestamp/sequence needed to compute it consistently.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
spec.md:87
- The spec uses “maximum output-token allowance” elsewhere (FR-112a, §11.2), but Assumption A-3 says “maximum output allowance”, which is ambiguous/inconsistent with the token-based budgeting terminology used in the rest of the PR.
| **A-3** | A frontier LLM with tool/function calling is available via API, with a caller-set maximum output allowance and per-call accounting for billable token categories. | §11.2 clarification. |
docs/adoption/clarification-playbook.md:44
- This example uses “maximum output allowance” and “cached” token category wording, but the rest of the spec set uses “maximum output-token allowance” and “cached-input”. Aligning the example text avoids introducing alternate terminology for the same concepts.
- Good: "Provider X via internal gateway Y; rate limits are exposed via response headers; requests enforce a maximum output allowance; responses report input/output/cached/reasoning tokens; versioned rate cards are stored with accounting events."
docs/operations/budget-and-stop-conditions.md:30
- Token category naming is inconsistent within the budget docs: the spec and observability checklist use “cached-input”, but this page uses “cached input”. Using a single term reduces ambiguity for implementers mapping provider fields into these categories.
- Token count by category when reported (input, output, cached input, reasoning).
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
| - **FR-113**: Spend tracking MUST account for every model call by every role. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates, and MUST surface what fraction of the reported total is estimated. | ||
| - **FR-112**: The Orchestrator MUST track cumulative settled LLM spend (in currency), outstanding cost reservations, and cumulative active wall-clock runtime across all runs of an evaluation. Active runtime MUST exclude operator-requested pauses and MUST NOT sum concurrent agent runtimes. The Orchestrator MUST invoke the FR-116a stop transition when settled spend reaches the spend cap or active runtime reaches the runtime cap. (US-5) | ||
| - **FR-112a**: Before dispatching any spend-creating model call, the budget governor MUST atomically reserve that call's conservative maximum cost against the spend cap, using the request's maximum output-token allowance and an immutable snapshot of the applicable provider/model rates. It MUST deny admission without contacting the provider when settled spend plus outstanding reservations plus the requested reservation would exceed the cap. On completion, failure, or cancellation it MUST settle the reservation exactly once against the billable actual or estimated cost and release any remainder. If a provider-reported charge exceeds its reservation, the governor MUST close new spend admission immediately and surface the discrepancy and any overrun as a degraded state. Budget admission enforces operator-authorized total spend; it is not the provider-rate pre-throttling prohibited by FR-105. | ||
| - **FR-113**: Spend tracking MUST record every model-call attempt by every role as a durable, idempotent accounting event. Each event MUST include a stable local call id; role and agent instance; provider and model; request, completion, or failure timestamps; outcome; input, output, cached-input, and reasoning token counts where reported; the immutable rate-card identity used for estimation; reserved cost; settled cost; and whether settled cost was provider-reported or estimated. A separately billed retry MUST have its own call id; replay of the same event MUST NOT count twice. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates. The surfaced estimated fraction MUST be estimated settled cost divided by total settled cost, not fraction of calls. |
There was a problem hiding this comment.
🟡 Changes recommended
Crash recovery, delayed-cost reconciliation, deterministic event ordering, and safe reset behavior remain underspecified.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
spec.md:733
- FR-115 orders events by settlement timestamp and a durable sequence id, but neither field is required in this accounting-event schema; completion/failure time is not necessarily settlement time. Implementations can therefore satisfy FR-113 while being unable to reproduce the required window or tie-break equal timestamps. Require both fields and define a total sequence order in the ledger contract.
- **FR-113**: Spend tracking MUST record every model-call attempt by every role as a durable, idempotent accounting event. Each event MUST include a stable local call id; role and agent instance; provider and model; request, completion, or failure timestamps; outcome; input, output, cached-input, and reasoning token counts where reported; the immutable rate-card identity used for estimation; reserved cost; settled cost; and whether settled cost was provider-reported or estimated. A separately billed retry MUST have its own call id; replay of the same event MUST NOT count twice. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates. The surfaced estimated fraction MUST be estimated settled cost divided by total settled cost, not fraction of calls.
- Files reviewed: 15/15 changed files
- Comments generated: 5
- Review effort level: Balanced
| - **FR-112**: The Orchestrator MUST track cumulative LLM spend (in currency) and cumulative wall-clock runtime across all runs of an evaluation, halting the fleet when either exceeds an operator-set cap. (US-5) | ||
| - **FR-113**: Spend tracking MUST account for every model call by every role. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates, and MUST surface what fraction of the reported total is estimated. | ||
| - **FR-112**: The Orchestrator MUST track cumulative settled LLM spend (in currency), outstanding cost reservations, and cumulative active wall-clock runtime across all runs of an evaluation. Active runtime MUST exclude operator-requested pauses and MUST NOT sum concurrent agent runtimes. The Orchestrator MUST invoke the FR-116a stop transition when settled spend reaches the spend cap or active runtime reaches the runtime cap. (US-5) | ||
| - **FR-112a**: Before dispatching any spend-creating model call, the budget governor MUST atomically reserve that call's conservative maximum cost against the spend cap, using the request's maximum output-token allowance and an immutable snapshot of the applicable provider/model rates. It MUST deny admission without contacting the provider when settled spend plus outstanding reservations plus the requested reservation would exceed the cap. On completion, failure, or cancellation it MUST settle the reservation exactly once against the billable actual or estimated cost and release any remainder. If a provider-reported charge exceeds its reservation, the governor MUST close new spend admission immediately and surface the discrepancy and any overrun as a degraded state. Budget admission enforces operator-authorized total spend; it is not the provider-rate pre-throttling prohibited by FR-105. |
| - **FR-126**: The configuration MUST be a single document (or a single directory treated as one) under version control alongside the evaluation's outputs. | ||
| - **FR-127**: Secrets (API keys, tokens, testbed credentials) MUST NOT be stored in the configuration document. The configuration references them; a separate non-version-controlled mechanism supplies them. | ||
| - **FR-128**: The Orchestrator MUST hot-reload `budget` and `rules` changes at runtime, in addition to the `fleet` hot-reload required by FR-009. | ||
| - **FR-128**: The Orchestrator MUST hot-reload `budget` and `rules` changes at runtime, in addition to the `fleet` hot-reload required by FR-009. A spend cap lowered below settled spend plus outstanding reservations, or a runtime cap lowered below active runtime, MUST immediately invoke FR-116a. Raising the cap that caused an FR-011 restart block MUST clear that block. Hot reload and restart MUST NOT implicitly reset cumulative spend, reservations, runtime, or accounting history; reset requires an explicit operator action that is durably recorded. |
Comment on lines
+732
to
+733
| - **FR-112a**: Before dispatching any spend-creating model call, the budget governor MUST atomically reserve that call's conservative maximum cost against the spend cap, using the request's maximum output-token allowance and an immutable snapshot of the applicable provider/model rates. It MUST deny admission without contacting the provider when settled spend plus outstanding reservations plus the requested reservation would exceed the cap. On completion, failure, or cancellation it MUST settle the reservation exactly once against the billable actual or estimated cost and release any remainder. If a provider-reported charge exceeds its reservation, the governor MUST close new spend admission immediately and surface the discrepancy and any overrun as a degraded state. Budget admission enforces operator-authorized total spend; it is not the provider-rate pre-throttling prohibited by FR-105. | ||
| - **FR-113**: Spend tracking MUST record every model-call attempt by every role as a durable, idempotent accounting event. Each event MUST include a stable local call id; role and agent instance; provider and model; request, completion, or failure timestamps; outcome; input, output, cached-input, and reasoning token counts where reported; the immutable rate-card identity used for estimation; reserved cost; settled cost; and whether settled cost was provider-reported or estimated. A separately billed retry MUST have its own call id; replay of the same event MUST NOT count twice. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates. The surfaced estimated fraction MUST be estimated settled cost divided by total settled cost, not fraction of calls. |
| Principles : III narrowed (work-reclamation only; session rotation per FR-118 carved out) | ||
| Sections changed : III | ||
| Version change : constitution 0.2.0 unchanged; spec 0.1.0 → 0.2.0 [MINOR: budget semantics strengthened] | ||
| Principles : V, VI, XI unchanged and re-validated against FR-112a, FR-113, FR-115, FR-116a |
|
|
||
| ``` | ||
| Version change : constitution 0.2.0 unchanged; spec 0.1.0 → 0.2.0 [MINOR: budget semantics strengthened] | ||
| Principles : V, VI, XI unchanged and re-validated against FR-112a, FR-113, FR-115, FR-116a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make Foundry's budget governor concurrency-safe and its accounting, trailing-yield, restart, and hot-reload behavior deterministic.
Closes #45.
Sync Impact Report
Failure case
FR-112 previously checked spend after calls were made. In a concurrent fleet, several agents could observe the same remaining budget and dispatch calls before the next governor poll, overshooting a supposed hard ceiling. The spec also left active-runtime calculation, per-call accounting identity, price-version handling, trailing spend-window boundaries, stop settlement, and cap hot reload ambiguous, allowing status/dashboard disagreement across otherwise conforming implementations.
FR changes
Constitutional alignment
No constitutional wording or version changes.
Documentation updates
Validation
git diff --checkOKfor all eleven principles./speckit.analyzecould not be run because this seed checkout does not include or install that command;plan.mdandtasks.mdare not present.