From f6dbe5713dc664a5b4a5062ca246cbb67350c474 Mon Sep 17 00:00:00 2001 From: Omar Santos Date: Thu, 10 Sep 2026 14:18:16 -0400 Subject: [PATCH 1/2] spec: make budget accounting concurrency-safe --- CHANGELOG.md | 3 +- GLOSSARY.md | 4 +- README.md | 3 +- constitution.md | 10 +- docs/adoption/clarification-playbook.md | 3 +- docs/adoption/integration-decisions.md | 6 +- docs/architecture/substrate-contracts.md | 29 +++++ docs/governance/sync-impact-reports.md | 18 ++- docs/operations/budget-and-stop-conditions.md | 112 ++++++++++++++++-- docs/operations/observability-checklist.md | 15 ++- docs/reference/fr-index.md | 12 +- docs/reference/open-questions-checklist.md | 2 +- docs/reference/principle-fr-matrix.md | 12 +- docs/reference/terminology-quick-card.md | 4 +- spec.md | 37 +++--- 15 files changed, 209 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b03722..7d246cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,7 +23,8 @@ When either changes at MINOR or above, a new entry below records the diff and th ### Changed -- *(none — all additions are net-new docs that cite, but do not modify, the canonical artifacts.)* +- `spec.md` 0.1.0 → 0.2.0 — make budget enforcement concurrency-safe and accounting deterministic (FR-011, FR-112, FR-112a, FR-113, FR-115, FR-116, FR-116a, FR-128; [#45](https://github.com/CiscoDevNet/foundry-security-spec/issues/45)). +- Budget operations, observability, substrate-contract, glossary, and reference documentation updated for atomic cost reservation, idempotent settlement, active-runtime semantics, deterministic trailing windows, and hot-reload/stop behavior. ### Removed diff --git a/GLOSSARY.md b/GLOSSARY.md index f1c83e4..723778a 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -26,11 +26,11 @@ A condensed quick-card version (one row per term, fewer terms) is at [`docs/refe | **Detection rule** | A reusable, versioned check for one vulnerability class, applied by the Detector to each function in scope. The rule corpus is an artifact independent of the agent code. | | **Rule-gap** | A record that an exploratory finding was confirmed `true-positive` and no detection rule would have produced it; the input to growing the rule corpus. See FR-042. | | **Coverage** | The degree to which the evaluation goals have been credibly attempted. | -| **Yield** | Severity-weighted confirmed findings per unit of spend, measured over a trailing window. | +| **Yield** | Severity-weighted confirmed findings per unit of settled spend, measured over a deterministic trailing spend window. | | **Work queue** | The shared, ordered list of tasks agents claim from. See [`spec.md` §8](spec.md#8-coordination-substrate). | | **Finding store** | The durable, fingerprint-indexed record of every finding at every lifecycle stage; internal, queryable by every role. Distinct from the issue tracker. | | **Coverage log** | The append-only record of which (area × technique) pairs the fleet has attempted; an audit trail, not a stop-list. See FR-046. | -| **Budget governor** | The substrate component that tracks spend, runtime, and trailing yield against operator caps and signals the Orchestrator to halt. See [`spec.md` §9.3–§9.4](spec.md#93-budget). | +| **Budget governor** | The substrate component that atomically reserves and settles model-call cost, tracks active runtime and trailing yield against operator caps, and signals the Orchestrator to halt. See [`spec.md` §9.3–§9.4](spec.md#93-budget). | | **Help request** | An operator-filed issue asking the fleet to perform a specific action; resolved by the Orchestrator's conversational facet. See FR-015. | | **Operator message** | An agent-authored, asynchronous, one-way note to the operator (blocker, request, feedback, or informational), deduplicated across the fleet. The agent→operator counterpart of a help request. See FR-102a. | | **Proof-of-concept (PoC)** | A self-contained, runnable artifact that demonstrates a finding's headline impact against the testbed. See FR-063. | diff --git a/README.md b/README.md index 481dd5b..b5ea89f 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ **An open specification for agentic AI security evaluation, from Cisco.** -[![Status](https://img.shields.io/badge/Status-Seed_v0.1.0-orange.svg)](CHANGELOG.md) +[![Status](https://img.shields.io/badge/Status-Seed_v0.2.0-orange.svg)](CHANGELOG.md) Cisco's Advanced Security Initiatives Group has built and operated an agentic security evaluation internally across several iterations and deployment models, and along the way accumulated a long list of design decisions that turned out to matter and a longer list that turned out not to. @@ -206,4 +206,3 @@ Original authors of the spec and constitution: - Theo Morales (@kh0rvus) - John Allbritten (@jallbrit) - diff --git a/constitution.md b/constitution.md index 8d0cf8b..a433051 100644 --- a/constitution.md +++ b/constitution.md @@ -1,13 +1,13 @@ diff --git a/docs/adoption/clarification-playbook.md b/docs/adoption/clarification-playbook.md index 5afce1b..ca5f529 100644 --- a/docs/adoption/clarification-playbook.md +++ b/docs/adoption/clarification-playbook.md @@ -41,8 +41,9 @@ The seed describes the *contract* each integration must satisfy. Your answer nam ### §11.2 LLM provider -- Good: "Provider X via internal gateway Y; rate limits are exposed via response headers." +- Good: "Provider X via internal gateway Y; rate limits are exposed via response headers; requests enforce a maximum output allowance; responses report input/output/cached/reasoning tokens; versioned rate cards are stored with accounting events." - Anti: "Multi-provider, we'll abstract over them." (Principle V — the provider is the rate arbiter — gets harder, not easier, behind an abstraction.) +- Anti: "Use the provider's current pricing page for all estimates." (Historical settled cost must not change when rates change.) ### §11.3 Datastore diff --git a/docs/adoption/integration-decisions.md b/docs/adoption/integration-decisions.md index 65c0df6..b095d55 100644 --- a/docs/adoption/integration-decisions.md +++ b/docs/adoption/integration-decisions.md @@ -39,7 +39,9 @@ Required capabilities the spec assumes: - The provider's responses include rate-limit headers or equivalent backoff signals. - Calls support timeouts at the call layer (separately from agent liveness). -- Token usage / cost is observable for budget enforcement (FR-112, FR-113). +- Every request supports a maximum output-token allowance so its conservative maximum cost can be reserved before dispatch (FR-112a). +- Token usage is observable by billable category; direct cost is preferable when the provider reports it (FR-113). +- Provider/model pricing can be captured as an immutable rate-card snapshot rather than a mutable “current price”. ## §11.3 Datastore @@ -54,6 +56,8 @@ Constraint: Principle XI — persist atomically. The finding store must never be **Anti-pattern:** "delete old, write new" persistence flows. Always write new, then atomically rename or commit (see [`spec.md` §8.6](../../spec.md#86-atomic-persistence)). +The same transaction boundary must protect budget admission: checking remaining spend and writing the reservation are one operation. A datastore that can atomically claim work but cannot atomically reserve budget does not satisfy FR-112a for a concurrent fleet. + ## §11.4 Vector search Constraint: optional per FR-023. If absent, the Indexer drops similarity search and the Variant-Hunter extension cannot be enabled (see [`extension-roles-when.md`](extension-roles-when.md)). diff --git a/docs/architecture/substrate-contracts.md b/docs/architecture/substrate-contracts.md index 978d187..7323f6f 100644 --- a/docs/architecture/substrate-contracts.md +++ b/docs/architecture/substrate-contracts.md @@ -182,6 +182,35 @@ provider_call(args) -> result --- +## Budget governance contract (§9.3–§9.4) + +```text +budget.admit(call_id, maximum_cost, rate_card_id) -> reservation | DENIED +budget.settle(call_id, billable_cost, reported_or_estimated) -> accounting_event +budget.snapshot() -> {settled, reserved, active_runtime, admission_state, stop_reason} +budget.evaluate_yield(window_size) -> {full, boundary, spend, points, yield} +budget.stop(reason) -> persisted_stop # idempotent +``` + +### Required guarantees + +| Guarantee | FR(s) | Principle | +|---|---|---| +| Concurrent admission cannot reserve more than the remaining operator-authored spend cap. | FR-112a | XI | +| Settlement and stop transitions are idempotent. | FR-113, FR-116a | XI | +| Rate-card identity is immutable for each reservation and settlement. | FR-112a, FR-113 | — | +| Budget admission does not impose a provider-rate or concurrency ceiling while authorized spend remains. | FR-112a, FR-105 | V | +| Every consumer derives the trailing window from the same ordered settled-event stream. | FR-115, FR-124 | VI, XI | + +### Anti-patterns + +- Read remaining budget, release the lock, then write a reservation — concurrent agents can all spend the same remainder. +- Dispatch before reservation commits — a denied call has already incurred cost. +- Recompute old settlements from the current provider price sheet — historical totals change under the operator. +- Use the budget path to cap calls per second — that is rate governance and contradicts Principle V. + +--- + ## Putting it together A reasonable substrate implementation is: diff --git a/docs/governance/sync-impact-reports.md b/docs/governance/sync-impact-reports.md index d3dae4a..d201b0f 100644 --- a/docs/governance/sync-impact-reports.md +++ b/docs/governance/sync-impact-reports.md @@ -102,7 +102,21 @@ If any check fails, the PR is not ready. These are preserved here as a navigable history. Git is authoritative; this archive is for convenience. -### 0.1.0 → 0.2.0 (current as of writing — see `constitution.md`) +### Spec 0.1.0 → 0.2.0 (current as of 2026-09-10 — constitution unchanged) + +``` +Version change : constitution 0.2.0 unchanged; spec 0.1.0 → 0.2.0 [MINOR: budget semantics strengthened] +Principles : V, VI, XI unchanged and re-validated against FR-112a, FR-113, FR-115, FR-116a +Sections changed : spec §1.6–§1.7, §3.2–§3.3, §4.4–§4.5, §5.1, §9.3–§9.4, §11.2, §12 +Templates needing update : n/a +Downstream re-check : spec.md ✓ README.md ✓ principle-fr-matrix.md ✓ plan.md n/a tasks.md n/a +Follow-up TODOs : none +Last sync : 2026-09-10 +``` + +**Why:** FR-011, FR-112–FR-116, and FR-128 previously allowed concurrent calls to overshoot a hard spend cap and left independent implementations free to compute different accounting totals, restart behavior, and trailing windows from the same events. The amendment adds atomic reservation, idempotent settlement and stopping, active-runtime semantics, immutable rate-card attribution, deterministic spend-window boundaries, and explicit hot-reload/reset behavior while preserving Principles V, VI, and XI. + +### Constitution 0.1.0 → 0.2.0 (archived 2026-05-04) ``` Version change : 0.1.0 → 0.2.0 [MINOR: III scope narrowed] @@ -118,7 +132,7 @@ Last sync : 2026-05-04 ### Future entries -When the constitution next changes, a new entry appears here, *above* the current entry, dated and tier-tagged. The constitution itself shows only the most recent block; this archive carries the trail. +When the report is next regenerated for a constitution change or a spec change that forces downstream re-validation, a new entry appears here, *above* the current entry, dated and tier-tagged. The constitution itself shows only the most recent block; this archive carries the trail. ## See also diff --git a/docs/operations/budget-and-stop-conditions.md b/docs/operations/budget-and-stop-conditions.md index 9546a37..9525a18 100644 --- a/docs/operations/budget-and-stop-conditions.md +++ b/docs/operations/budget-and-stop-conditions.md @@ -12,51 +12,101 @@ The conjunction "yield low AND coverage complete AND minimum runtime AND trailin The Orchestrator must track and surface: -- [ ] Cumulative LLM spend across all runs of an evaluation, in currency. -- [ ] Cumulative wall-clock runtime across all runs. +- [ ] Cumulative settled LLM spend across all runs of an evaluation, in currency. +- [ ] Outstanding cost reservations for admitted calls that have not settled. +- [ ] Cumulative active wall-clock runtime across all runs, excluding operator-requested pauses and without multiplying time by fleet width. - [ ] Per-call cost attribution (FR-113), including which fraction is provider-reported vs estimated. +- [ ] Whether spend admission is open or closed and the durable reason for the latest stop decision. ### Cost attribution For every model call, record: -- Role (`detector`, `triager`, etc.). +- Stable local call id; provider request id when one is available. +- Role (`detector`, `triager`, etc.) and agent instance. - Tool name (if applicable). -- Provider (in case you have multiple — though Principle V cautions against multi-provider abstraction). -- Token count (input, output). -- Reported cost OR estimated cost from token rates. +- Provider and model (in case you have multiple — though Principle V cautions against multi-provider abstraction). +- Request, completion, or failure timestamps and outcome. +- Token count by category when reported (input, output, cached input, reasoning). +- Immutable rate-card identity and reserved maximum cost. +- Reported cost OR estimated settled cost from token rates. - Whether reported or estimated. -The "fraction estimated" surfacing requirement (FR-113) exists because internal estimates drift from provider invoices. Surfacing the fraction lets operators sanity-check. +Record a separately billed retry as a new call. Make ingestion idempotent on the local call id so replaying one accounting event cannot charge the evaluation twice. + +The "fraction estimated" surfacing requirement (FR-113) exists because internal estimates drift from provider invoices. Compute it by currency value — `estimated settled cost / total settled cost` — rather than by number of calls. A thousand cheap estimated calls should not obscure one expensive provider-reported call, or vice versa. + +### Admission, reservation, and settlement + +A check performed only after provider calls return is not a hard spend ceiling in a concurrent fleet. Every spend-creating call therefore passes through one atomic admission path: + +```text +admit(call_id, maximum_cost): + transaction: + return prior result if call_id already exists + deny if admission is closed + deny if cap is set and settled + reserved + maximum_cost > cap + persist reservation(call_id, maximum_cost, rate_card_id) + reserved += maximum_cost + dispatch provider call + +settle(call_id, actual_or_estimated_cost): + transaction: + return prior result if call_id is already settled + reserved -= reservation(call_id) + settled += actual_or_estimated_cost + persist settlement exactly once + if actual_or_estimated_cost > reservation(call_id): + close admission and surface a degraded-state discrepancy +``` + +The maximum is computed from the complete request, including its maximum output-token allowance, every billable token category, and an immutable snapshot of the applicable provider/model rates. If the spend cap is unset, the same ledger path records and settles the reservation but does not reject for insufficient budget. + +Reservation is budget admission, not rate governance. It answers “has the operator authorized enough total spend for this call?” It does not answer “how fast may we call the provider?” and therefore does not conflict with Principle V or FR-105. + +### Active runtime + +Runtime is elapsed evaluation time, not aggregate agent-process time: + +- Start or resume advances the active-runtime clock. +- An operator-requested pause freezes it. +- Ten agents running concurrently for one hour add one hour, not ten. +- Restarting a run continues the same evaluation's cumulative clock unless the operator explicitly resets it. +- Reaching the runtime cap closes spend admission and begins shutdown within one governor polling interval; already-reserved calls still settle. ### Budget caps - [ ] Cap configurable for both spend and runtime ([FR-112](../../spec.md#93-budget)). - [ ] Default unset (unlimited) ([FR-114](../../spec.md#93-budget)). - [ ] Pre-flight warning when both are unset ([FR-114](../../spec.md#93-budget)). -- [ ] Hard halt when either is exceeded. +- [ ] Atomic denial before a new call would reserve beyond the spend cap ([FR-112a](../../spec.md#93-budget)). +- [ ] Spend admission closes and shutdown begins when either cap is reached. ### Refusing a re-run after a hit cap -[FR-011](../../spec.md#51-orchestrator): the Orchestrator MUST refuse to start a new evaluation run if a previous run hit a hard budget cap and the cap has not been raised. The error message must say which cap and how to reset. +[FR-011](../../spec.md#51-orchestrator): the Orchestrator MUST refuse to start a new evaluation run after a hard-cap stop unless the exhausted cap has been raised enough to admit work or the operator explicitly resets the corresponding cumulative accounting. The error message must identify the cap, show current settled/reserved spend or active runtime, and say how to raise or reset it. -This is to prevent the operator from blindly restarting a runaway run. +This prevents the operator from blindly restarting a runaway run while preserving an explicit, auditable recovery path. --- ## Yield definition (§9.4) -[FR-115](../../spec.md#94-yield-auto-stop): trailing yield = severity-weighted, exploited-multiplied confirmed findings divided by spend, over a trailing spend window. +[FR-115](../../spec.md#94-yield-auto-stop): trailing yield = severity-weighted, exploited-multiplied confirmed findings divided by settled spend, over a trailing spend window. ``` yield(t) = ΣᵢweightedSeverity(findingᵢ) × exploitedMultiplier(findingᵢ) ÷ - spend in last W + settled spend in trailing window W ``` Where: -- The sum runs over confirmed findings whose `true-positive` confirmation timestamp falls in the trailing window. +- The accounting events are ordered chronologically by settlement timestamp, with a durable sequence id breaking ties. +- The window is the newest suffix whose spend is at least `W` and would be below `W` if its oldest event were removed. +- The sum runs over confirmed findings whose `true-positive` confirmation timestamp falls between the oldest included event's settlement timestamp and now. +- Before cumulative settled spend reaches `W`, the window is not full and yield cannot trigger auto-stop. +- Because calls are indivisible, the denominator may exceed `W` by the oldest included call's cost; consumers must use that actual denominator. - `weightedSeverity` follows a geometric scale (FR-117 rationale): roughly 3.15× per tier, calibrated against multi-year bug-bounty payout ratios. - `exploitedMultiplier` is 2× for findings with `exploited=true`. @@ -138,6 +188,25 @@ Eight hours in. Yield 0.04 (just under threshold). Coverage 95% complete (a few --- +## Stop transition and hot reload + +Every hard-cap or low-yield decision uses one idempotent state transition: + +1. Atomically close admission of new spend-creating calls. +2. Persist one stop reason and a snapshot of settled spend, reservations, active runtime, coverage, and yield. +3. Signal the Orchestrator to drain. +4. Allow already-reserved calls to settle; do not admit cleanup calls that create new LLM spend. +5. Perform bounded non-LLM cleanup, release claims, and stop. + +Budget hot reload keeps the same accounting history: + +- Lowering a spend cap below `settled + reserved`, or a runtime cap below active runtime, invokes the stop transition immediately. +- Raising the cap that caused the prior hard stop clears the FR-011 restart block. +- Removing a cap makes that dimension unlimited but does not erase what was consumed. +- Resetting cumulative accounting is a separate explicit operator action and must be durably recorded. + +--- + ## Yield-window and minimum-runtime tuning ### Trailing window @@ -190,8 +259,25 @@ Forbidden: | Setting coverage-complete to suppress auto-stop. | Defeats the conjunction; reproduces the failure Principle VI exists for. | | Linear severity scale. | Low-severity volume dominates yield economically. | | Per-call cost without role attribution. | Cannot identify where spend is going (FR-123 violated). | +| Checking the spend cap only after calls return. | Concurrent in-flight calls can overshoot the supposed hard ceiling. | +| Counting calls rather than using idempotent call ids. | Retries and replayed events can charge the same work twice. | +| Treating runtime as the sum of agent runtimes. | Increasing fleet width burns a time budget faster without increasing elapsed evaluation time. | +| Recomputing historical estimates with today's prices. | Identical accounting history changes when a provider updates its rates. | | Pre-throttling LLM calls below provider limit "to control budget". | Forbidden — Principle V. Use the budget cap, not the rate cap, to control spend. | +## Conformance cases + +| Case | Required assertion | +|---|---| +| Twenty calls concurrently request reservations whose sum exceeds the remaining cap. | The serialized admitted reservations fit within the cap; every denied call is rejected before provider dispatch. | +| The same settlement event is delivered twice. | Settled spend changes once. | +| A retry is separately billed by the provider. | The retry has a distinct local call id and both billable attempts appear once. | +| The evaluation is paused for one hour. | Active runtime does not advance during the pause. | +| Ten agents run for one hour. | Active runtime advances by one hour, not ten. | +| Two consumers replay the same ordered accounting and finding events. | Both select the same trailing suffix and compute the same yield. | +| A cap is lowered below settled spend plus reservations. | Admission closes immediately, one stop reason is persisted, and repeated checks do not create additional stop records. | +| A cap is raised after a cap-triggered stop. | FR-011's restart block clears while cumulative accounting remains unchanged. | + ## See also - [`observability-checklist.md`](observability-checklist.md) — surfacing yield, coverage, and budget on the dashboard. diff --git a/docs/operations/observability-checklist.md b/docs/operations/observability-checklist.md index 11acfaf..9e16216 100644 --- a/docs/operations/observability-checklist.md +++ b/docs/operations/observability-checklist.md @@ -40,9 +40,13 @@ The Operator dashboard MUST show, at minimum: ### Budget against caps -- [ ] LLM spend consumed (USD or local equivalent). -- [ ] Wall-clock runtime consumed. +- [ ] Settled LLM spend consumed (USD or local equivalent). +- [ ] Outstanding reserved spend for admitted calls. +- [ ] Active wall-clock runtime consumed, excluding operator-requested pauses. - [ ] Caps (configured value). +- [ ] Spend admission state (open / closed) and durable stop reason when closed. +- [ ] Provider-reported vs estimated settled cost, with the estimated fraction weighted by currency value. +- [ ] Reservation discrepancies where a provider-reported charge exceeded its reserved maximum. **Source:** the budget governor ([§9.3](../../spec.md#93-budget)). @@ -51,6 +55,7 @@ The Operator dashboard MUST show, at minimum: - [ ] Yield value (severity-weighted findings per spend). - [ ] Configured threshold. - [ ] Coverage-complete flag (the conjunction prerequisite). +- [ ] Whether the trailing spend window is full, its actual settled-spend denominator, and its oldest included event. **Source:** [§9.4 / FR-115–FR-117](../../spec.md#94-yield-auto-stop). Display the threshold as a horizontal line on the yield chart so operators can see proximity. @@ -88,18 +93,20 @@ The Operator dashboard MUST show, at minimum: ## Cost & token rollups (FR-123) - [ ] Per-role cost. -- [ ] Per-role token usage. +- [ ] Per-role token usage, separated into input, output, cached-input, and reasoning categories where reported. +- [ ] Rate-card identity and reported/estimated status for every settled call. - [ ] Tool-usage histogram. - [ ] Granularity sufficient for an operator (or Self-Improver) to identify where spend is going. **Implementation patterns:** -- Tags / labels on every LLM call: `role`, `instance`, `tool`, `target`. Aggregate on the dashboard. +- Tags / labels on every LLM call: `call_id`, `role`, `instance`, `tool`, `target`, `provider`, `model`, `rate_card_id`. Aggregate on the dashboard. - Daily / weekly cost reports filed automatically into the operator's tracker if budget approaching cap. ## Dashboard / source-of-truth coherence (FR-124) - [ ] The status query (FR-008) and the dashboard agree with each other and with the substrate's actual contents. +- [ ] Settled spend, reserved spend, active runtime, admission state, and stop reason come from the same authoritative accounting view. **This is the single most-important invariant in observability.** A dashboard that reads a stale or differently-computed view of the substrate produces operator decisions based on fiction. diff --git a/docs/reference/fr-index.md b/docs/reference/fr-index.md index 3c05683..b357bce 100644 --- a/docs/reference/fr-index.md +++ b/docs/reference/fr-index.md @@ -30,7 +30,7 @@ Spec link: [`spec.md` §5.1 Orchestrator](../../spec.md#51-orchestrator) | **FR-008** | The Orchestrator MUST expose a status query that reports, for each agent: role, instance index, alive/dead, current claim if any, last heartbeat age, restart count. (US-2) | | **FR-009** | The Orchestrator MUST hot-reload changes to fleet composition (role counts) from configuration without a full restart, reconciling by spawning shortfall and gracefully draining... | | **FR-010** | The Orchestrator SHOULD pre-flight check external dependencies (LLM provider reachable, issue tracker credentials valid, testbed reachable if configured) and report all failures... | -| **FR-011** | The Orchestrator MUST refuse to start a new evaluation run if a previous run hit a hard budget cap and the cap has not been raised, with a message stating which cap and how to r... | +| **FR-011** | Refuse restart after a hard-cap stop until the exhausted cap is raised enough to admit work or its cumulative accounting is explicitly reset; report current usage and recovery steps. | | **FR-012** | The Orchestrator MUST NOT itself perform detection, triage, validation, or reporting. | | **FR-013** | The Orchestrator MUST answer free-form operator questions about evaluation state, grounded in the actual substrate contents (not the model's general knowledge), citing the recor... | | **FR-014** | The Orchestrator MUST accept operator-submitted tasks and place them on the work queue at the operator's chosen priority. (US-12) | @@ -289,8 +289,9 @@ Spec link: [`spec.md` §9.3 Budget](../../spec.md#93-budget) | FR | Summary | |---|---| -| **FR-112** | The Orchestrator MUST track cumulative LLM spend (in currency) and cumulative wall-clock runtime across all runs of an evaluation, halting the fleet when either exceeds an opera... | -| **FR-113** | Spend tracking MUST account for every model call by every role. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates... | +| **FR-112** | Track cumulative settled spend, outstanding reservations, and active evaluation runtime across runs; invoke the stop transition when settled spend or active runtime reaches its cap. | +| **FR-112a** | Atomically reserve a model call's conservative maximum cost before provider dispatch, deny calls that do not fit, and settle each reservation exactly once. | +| **FR-113** | Record every model-call attempt as a durable, idempotent accounting event with attribution, token categories, immutable pricing identity, reservation, settlement, and reported/estimated status. | | **FR-114** | Budget caps default to unset (unlimited). The pre-flight check (FR-010) SHOULD warn when both are unset. | ## §9.4 Yield auto-stop @@ -299,8 +300,9 @@ Spec link: [`spec.md` §9.4 Yield auto-stop](../../spec.md#94-yield-auto-stop) | FR | Summary | |---|---| -| **FR-115** | The system MUST compute trailing yield: confirmed findings, weighted by severity and by `exploited` status, divided by spend, over a trailing spend window. | +| **FR-115** | Compute trailing yield over a deterministic chronological suffix of settled accounting events, using its actual settled spend and confirmed findings within its time boundary. | | **FR-116** | The Orchestrator MUST halt the fleet when trailing yield falls below an operator-set threshold (US-4), but ONLY when all of: (a) at least one full trailing window of spend has a... | +| **FR-116a** | Atomically close spend admission on stop, settle already-reserved calls, permit only bounded non-LLM cleanup, and persist one idempotent stop reason and budget snapshot. | | **FR-117** | Severity weights, the exploited multiplier, the trailing window size, the minimum runtime, and the threshold MUST be operator-configurable. The seed does not prescribe values. | ## §9.5 Agent lifecycle limits @@ -334,7 +336,7 @@ Spec link: [`spec.md` §12. Configuration Model](../../spec.md#12-configuration- |---|---| | **FR-126** | The configuration MUST be a single document (or a single directory treated as one) under version control alongside the evaluation's outputs. | | **FR-127** | Secrets (API keys, tokens, testbed credentials) MUST NOT be stored in the configuration document. The configuration references them; a separate non-version-controlled mechanism... | -| **FR-128** | The Orchestrator MUST hot-reload `budget` and `rules` changes at runtime, in addition to the `fleet` hot-reload required by FR-009. | +| **FR-128** | Hot-reload budget and rules; immediately stop when a lowered cap is already exhausted, clear the matching restart block when raised, and never reset accounting implicitly. | | **FR-128a** | Changes to `target`, `sandbox`, and `integrations` MAY require a restart. | | **FR-129** | A configuration with unfilled required fields MUST fail FR-001 validation with a message naming each missing field. | diff --git a/docs/reference/open-questions-checklist.md b/docs/reference/open-questions-checklist.md index d0c9aad..711217c 100644 --- a/docs/reference/open-questions-checklist.md +++ b/docs/reference/open-questions-checklist.md @@ -60,7 +60,7 @@ For deeper guidance on each, see [`../adoption/clarification-playbook.md`](../ad - [ ] **§9.1 / §11.6** — Sandbox enforcement mechanism (gateway, host firewall, security groups, network policy, runtime isolation, etc.). - [ ] **§10** — Dashboard delivery mechanism (web UI, terminal, static, panels on existing observability stack). - [ ] **§11.1** — VCS host & issue tracker. -- [ ] **§11.2** — LLM provider, models, tiering. +- [ ] **§11.2** — LLM provider, models, tiering, billable token categories/direct cost, and immutable rate-card source. - [ ] **§11.3** — Datastore. - [ ] **§11.5** — Deployment topology; single- vs multi-tenant. - [ ] **§11.7** — Authentication model to VCS/tracker. diff --git a/docs/reference/principle-fr-matrix.md b/docs/reference/principle-fr-matrix.md index 6fcbef1..94b2897 100644 --- a/docs/reference/principle-fr-matrix.md +++ b/docs/reference/principle-fr-matrix.md @@ -23,13 +23,13 @@ If this matrix and the spec disagree, the spec wins; this page is updated. | II | [Surface Only What Survives](../../constitution.md#ii-surface-only-what-survives) | FR-044, FR-057, FR-079 | FR-042 (rule-gap), FR-045 (dedup), FR-078 (publish true-positive reports), FR-086 (retain rejected), FR-090–FR-091 (fingerprint), FR-092 (label set) | OK | | III | [Liveness By Heartbeat, Never By Clock](../../constitution.md#iii-liveness-by-heartbeat-never-by-clock) | FR-005, FR-100, FR-101 | FR-007 (crash-loop backoff), FR-118 (session rotation, *not* liveness) | OK | | IV | [Claims Are Atomic And Mortal](../../constitution.md#iv-claims-are-atomic-and-mortal) | FR-095, FR-096 | FR-097 (auto-block on N releases), FR-098 (operator-/agent-writable), FR-099 (stable ids) | OK | -| V | [The Provider Is The Rate Arbiter](../../constitution.md#v-the-provider-is-the-rate-arbiter) | FR-105, FR-106 | FR-113 (cost/token tracking) | OK | -| VI | [Coverage Before Yield](../../constitution.md#vi-coverage-before-yield) | FR-115, FR-116, FR-117 | FR-067–FR-071 (Coverage-Guide), FR-114 (default unset) | OK | +| V | [The Provider Is The Rate Arbiter](../../constitution.md#v-the-provider-is-the-rate-arbiter) | FR-105, FR-106 | FR-112a (operator-budget admission), FR-113 (cost/token tracking) | OK | +| VI | [Coverage Before Yield](../../constitution.md#vi-coverage-before-yield) | FR-115, FR-116, FR-117 | FR-067–FR-071 (Coverage-Guide), FR-114 (default unset), FR-116a (atomic stop transition) | OK | | VII | [Exploited Means Demonstrated](../../constitution.md#vii-exploited-means-demonstrated) | FR-060, FR-061, FR-089 | FR-062–FR-064 (Validator behavior) | OK | | VIII | [Fingerprints Are Stable Under Edit](../../constitution.md#viii-fingerprints-are-stable-under-edit) | FR-090, FR-091 | FR-045 (dedup keys on fingerprint), FR-080 (Reporter dedup), FR-058 (cross-run inheritance) | OK | | IX | [Sandbox By Infrastructure, Not By Prompt](../../constitution.md#ix-sandbox-by-infrastructure-not-by-prompt) | FR-107, FR-108 | FR-109 (operator informed of pivot points), FR-110, FR-111 (hard rules as defense-in-depth) | OK | | X | [The Operator Outranks Every Agent](../../constitution.md#x-the-operator-outranks-every-agent) | FR-014, FR-016, FR-018, FR-102 | FR-013 (grounded answers), FR-015 (help requests), FR-110, FR-119a (no busywork) | OK | -| XI | [Persist Atomically](../../constitution.md#xi-persist-atomically) | FR-106a | FR-025 (Indexer instance), FR-074 (Coverage-Guide checklist instance) | OK | +| XI | [Persist Atomically](../../constitution.md#xi-persist-atomically) | FR-106a | FR-025 (Indexer instance), FR-074 (Coverage-Guide checklist instance), FR-112a/FR-113 (budget reservation and settlement), FR-116a (stop transition) | OK | All eleven principles have at least one Primary FR. **No GAP rows.** Status: matrix passes. @@ -53,11 +53,11 @@ FR-095 (atomicity) and FR-096 (mortality). Both are required; either alone repro ### V — The Provider Is The Rate Arbiter -FR-105 forbids internal pre-throttling below the provider's actual limit. FR-106 requires shared backoff state across the fleet. +FR-105 forbids internal pre-throttling below the provider's actual limit. FR-106 requires shared backoff state across the fleet. FR-112a separately admits calls against the operator's total-spend authorization; it does not cap provider throughput while authorized budget remains. ### VI — Coverage Before Yield -FR-116 is the conjunction (yield low AND coverage complete AND minimum runtime AND trailing window full). FR-115 defines yield. FR-117 makes the parameters operator-configurable. +FR-116 is the conjunction (yield low AND coverage complete AND minimum runtime AND trailing window full). FR-115 defines a deterministic spend-window calculation. FR-117 makes the parameters operator-configurable, and FR-116a makes the resulting stop transition idempotent. ### VII — Exploited Means Demonstrated @@ -77,7 +77,7 @@ FR-014 (operator submits tasks at chosen priority), FR-016 (operator steers runn ### XI — Persist Atomically -FR-106a is the general rule. FR-025 and FR-074 are instances of it (Indexer index, Coverage-Guide checklist). Removing FR-106a would not by itself remove FR-025 / FR-074, but the general rule is required so future persisted artifacts inherit the guarantee. +FR-106a is the general rule. FR-025 and FR-074 are instances of it (Indexer index, Coverage-Guide checklist); FR-112a, FR-113, and FR-116a apply it to budget admission, settlement, and stopping. Removing FR-106a would not by itself remove these instances, but the general rule is required so future persisted artifacts inherit the guarantee. ## Procedure: regenerate this matrix diff --git a/docs/reference/terminology-quick-card.md b/docs/reference/terminology-quick-card.md index aedf2a8..19a7698 100644 --- a/docs/reference/terminology-quick-card.md +++ b/docs/reference/terminology-quick-card.md @@ -48,9 +48,9 @@ Operator ──► Orchestrator ──► Substrate ──► Roles (8 core, 5 e | **Work queue** | Shared, ordered list of tasks agents claim from (atomic, mortal). | | **Finding store** | Durable, fingerprint-indexed record of every finding. Internal, distinct from the issue tracker. | | **Coverage log** | Append-only audit trail of (area × technique) pairs attempted. | -| **Budget governor** | Tracks spend, runtime, and trailing yield against operator caps. | +| **Budget governor** | Atomically reserves and settles model-call cost; tracks active runtime and trailing yield against operator caps. | | **Coverage** | Degree to which evaluation goals were credibly attempted. | -| **Yield** | Severity-weighted confirmed findings per unit spend, over a trailing window. | +| **Yield** | Severity-weighted confirmed findings per unit settled spend, over a deterministic trailing spend window. | | **Claim** | Exclusive, crash-safe hold on a unit of work. | | **Heartbeat** | Liveness signal an agent emits on its own execution lane. | diff --git a/spec.md b/spec.md index c12ed66..37bbab4 100644 --- a/spec.md +++ b/spec.md @@ -3,7 +3,7 @@ | Field | Value | |---|---| | **Status** | `SEED` | -| **Version** | 0.1.0 | +| **Version** | 0.2.0 | | **Intended use** | Input to `/speckit.clarify`. This is not a finished specification. | | **Companion files** | `constitution.md`; see `README.md` for usage | @@ -74,6 +74,7 @@ These are technology-agnostic, measurable outcomes a Foundry implementation must | **SC-007** | An agent inside the sandbox, given root and an explicit instruction to do so, cannot open a connection to a host outside the configured allowlist. | US-6, FR-107, Constitution IX | | **SC-008** | Operator status query and dashboard report identical fleet state, finding counts, and budget figures at the same instant. | US-2, FR-008, FR-120, FR-124 | | **SC-009** | For any published finding, the full provenance chain (detection technique → triage transcript → validation attempt → report render) is reconstructable from logs alone. | NFR-007, FR-122 | +| **SC-010** | With concurrent model calls competing for less remaining budget than their combined maximum cost, admitted reservations never exceed the remaining cap; calls denied admission never reach the provider; and status and dashboard show identical settled-spend and reserved-spend totals. | US-5, FR-112, FR-112a, FR-120, FR-124, Constitution XI | ### 1.7 Assumptions @@ -83,7 +84,7 @@ The seed is written assuming the following hold. Each is either confirmed or ove |---|---|---| | **A-1** | The operator is authorized to evaluate the target and to probe the testbed; authorization is established outside the system. | Out of scope (§14). | | **A-2** | The operator has read access to the target's source code. | §1.5 clarification; black-box mode is out of scope (§14). | -| **A-3** | A frontier LLM with tool/function calling is available via API, with per-call token accounting. | §11.2 clarification. | +| **A-3** | A frontier LLM with tool/function calling is available via API, with a caller-set maximum output allowance and per-call accounting for billable token categories. | §11.2 clarification. | | **A-4** | The deployment environment can enforce network-egress and filesystem-write boundaries below the agent process (i.e., not by prompt alone). | §9.1 / §11.6 clarification; "trust the prompt" is rejected by FR-107. | | **A-5** | The operator is a security engineer comfortable reading code, editing a config file, and interpreting a vulnerability writeup; the system does not target non-technical users. | Persona table §3.1. | | **A-6** | A single evaluation's working set (index, finding store, queue, logs) fits in storage reachable by every agent in the fleet. | §11.3 / §11.5 clarifications. | @@ -115,11 +116,11 @@ The seed is written assuming the following hold. Each is either confirmed or ove | **Detection rule** | A reusable, versioned check for one vulnerability class, applied by the Detector to each function in scope. The rule corpus is an artifact independent of the agent code. | | **Rule-gap** | A record that an exploratory finding was confirmed `true-positive` and no detection rule would have produced it; the input to growing the rule corpus. See FR-042. | | **Coverage** | The degree to which the evaluation goals have been credibly attempted. | -| **Yield** | Severity-weighted confirmed findings per unit of spend, measured over a trailing window. | +| **Yield** | Severity-weighted confirmed findings per unit of settled spend, measured over a deterministic trailing spend window. | | **Work queue** | The shared, ordered list of tasks agents claim from. See §8. | | **Finding store** | The durable, fingerprint-indexed record of every finding at every lifecycle stage; internal, queryable by every role. Distinct from the issue tracker. | | **Coverage log** | The append-only record of which (area × technique) pairs the fleet has attempted; an audit trail, not a stop-list. See FR-046. | -| **Budget governor** | The substrate component that tracks spend, runtime, and trailing yield against operator caps and signals the Orchestrator to halt. See §9.3–§9.4. | +| **Budget governor** | The substrate component that atomically reserves and settles model-call cost, tracks active runtime and trailing yield against operator caps, and signals the Orchestrator to halt. See §9.3–§9.4. | | **Help request** | An operator-filed issue asking the fleet to perform a specific action; resolved by the Orchestrator's conversational facet. See FR-015. | | **Operator message** | An agent-authored, asynchronous, one-way note to the operator (blocker, request, feedback, or informational), deduplicated across the fleet. The agent→operator counterpart of a help request. See FR-102a. | | **Proof-of-concept (PoC)** | A self-contained, runnable artifact that demonstrates a finding's headline impact against the testbed. See FR-063. | @@ -151,7 +152,7 @@ The system MUST support the following user stories. Each is satisfied by one or | **US-2** | P1 | Operator | See, at any moment, what every agent is working on, what it has found, and what it is blocked on | I can intervene when something goes wrong instead of discovering it at the end | `status` and dashboard each enumerate every live agent with its current claim. | | **US-3** | P2 | Operator | Ask the running system a free-form question ("why was finding #14 closed?", "has anyone looked at the auth module?") and get an answer grounded in the evaluation's actual state | I do not have to read raw logs | A query about a known finding's verdict returns the recorded reasoning with a citation to the store record. | | **US-4** | P1 | Operator | Be told when the evaluation is done, and why | I am not guessing when to stop paying for compute | SC-006. | -| **US-5** | P1 | Operator | Set a hard ceiling on spend and/or wall-clock time | A runaway evaluation cannot cause an unbounded bill | With a low spend cap set, fleet halts within one polling interval of the cap being crossed; FR-011 refuses restart. | +| **US-5** | P1 | Operator | Set a hard ceiling on spend and/or wall-clock time | A runaway evaluation cannot cause an unbounded bill | SC-010; with a low time cap set, spend admission closes and the fleet begins draining within one polling interval of the cap being reached; FR-011 refuses restart. | | **US-6** | P1 | Operator | Constrain what the agent fleet can reach on the network and modify on disk | Agents probing a live deployment cannot accidentally reach production, the internet, or their own configuration | SC-007. | | **US-7** | P1 | Reviewer | Receive only findings that have passed a structural evidence check, with the evidence attached | I am not triaging the model's hallucinations | SC-002; every published issue carries resolvable §7.3 citations. | | **US-8** | P2 | Reviewer | See which findings were actually demonstrated against a running system, distinct from those only argued from code | I can prioritize what is proven over what is plausible | SC-003. Depends on a testbed (§11.12). | @@ -183,6 +184,8 @@ The system MUST support the following user stories. Each is satisfied by one or | Operator overrides an automated verdict | Override applied and recorded; system does not refuse. | NFR-009, FR-018, Constitution Governance | | No testbed configured | Validator degrades to PoC-only; `exploited` is never set. | FR-066 | | Budget caps both unset | Pre-flight warns; run is unbounded by spend/time, bounded only by coverage∧yield. | FR-114, FR-116 | +| Concurrent calls compete for the remaining spend budget | The budget governor atomically admits only reservations that fit; denied calls do not reach the provider. | FR-112a, SC-010, Constitution XI | +| Provider reports a billable charge above the reserved maximum | New spend admission closes immediately; the discrepancy and any resulting overrun are surfaced as a degraded state. | FR-112a, FR-125 | | Agent exceeds soft session limit | Steered to wrap up and release claims. | FR-118 | | Agent exceeds hard session limit | Process terminated; fresh instance spawned in slot; any claim still held is released by FR-096, not re-queued by the rotation itself. | FR-118, FR-096, Constitution III | | Agent hits a human-only blocker | Agent posts an operator message (`kind=blocker`/`request`), releases any held claim, continues with other work; does not wait for a reply. | FR-102a | @@ -268,7 +271,7 @@ Not agents; the machinery every agent uses. Specified as **behavior** in §8-§1 | **Work queue** | Ordered tasks; atomic claim; crash-safe release on holder death; bounded retry with auto-block. | | **Finding store** | Durable record of every finding at every lifecycle stage; fingerprint-indexed; queryable by every role. | | **Sandbox** | Network egress constrained to an allowlist; filesystem write constrained to designated paths; survives agent root. | -| **Budget governor** | Tracks spend and runtime against operator-set caps; computes trailing yield; halts the fleet on cap or on yield-below-threshold once coverage is complete. | +| **Budget governor** | Atomically reserves and settles spend against operator-set caps; tracks active runtime; computes trailing yield; halts the fleet on cap or on yield-below-threshold once coverage is complete. | | **Dashboard** | Operator-facing live view of fleet state, findings, coverage, budget, yield. | ### 4.5 What an evaluation looks like end to end @@ -282,7 +285,7 @@ Not agents; the machinery every agent uses. Specified as **behavior** in §8-§1 7. Validator attempts clean-room reproduction; marks findings `exploited` or records why not. 8. Reporter writes up each confirmed finding as it lands and maintains the rollup. 9. Coverage-Guide watches the finding store and work queue, checks off goals as they are credibly attempted, eventually marks coverage complete. -10. Budget governor watches yield. Once coverage is complete **and** trailing yield is below threshold (or a hard cap is hit), it signals the Orchestrator, which drains the fleet and stops. +10. Budget governor admits model calls through atomic cost reservations and watches runtime and yield. Once coverage is complete **and** trailing yield is below threshold (or a hard cap is reached), it closes spend admission and signals the Orchestrator, which settles already-reserved calls, drains the fleet, and stops. 11. Throughout, the Operator can ask the Orchestrator questions, queue tasks, steer agents, adjust scope, or stop early. *Rationale: step 3's "index first, then spawn the rest" gate exists because every downstream role's quality depends on being able to ask "who calls this function" and "where is this symbol defined". Early versions that ran detection in parallel with indexing produced findings that could not be investigated because the investigation tools had nothing to query. The gate costs minutes and saves hours. Whether the Cartographer ALSO gates the fleet is a §5.3 clarification: its outputs improve every downstream role's reasoning, but unlike the index they are not strictly required for those roles to function, and on large targets the map can take materially longer to produce. The seed's default is no gate, with roles reading whatever map exists when they need it; the clarification lays out when a gate or a soft gate is the better choice.* @@ -330,7 +333,7 @@ Throughout this section, "MUST" means the seed authors consider the requirement - **FR-008**: The Orchestrator MUST expose a status query that reports, for each agent: role, instance index, alive/dead, current claim if any, last heartbeat age, restart count. (US-2) - **FR-009**: The Orchestrator MUST hot-reload changes to fleet composition (role counts) from configuration without a full restart, reconciling by spawning shortfall and gracefully draining surplus. - **FR-010**: The Orchestrator SHOULD pre-flight check external dependencies (LLM provider reachable, issue tracker credentials valid, testbed reachable if configured) and report all failures at once rather than failing on the first. -- **FR-011**: The Orchestrator MUST refuse to start a new evaluation run if a previous run hit a hard budget cap and the cap has not been raised, with a message stating which cap and how to reset. +- **FR-011**: The Orchestrator MUST refuse to start a new evaluation run after a hard-cap stop unless the exhausted cap has been raised enough to admit work or the operator has explicitly reset the corresponding cumulative accounting under FR-128. The refusal MUST identify the cap, report its current settled/reserved spend or active runtime, and explain how to raise or reset it. - **FR-012**: The Orchestrator MUST NOT itself perform detection, triage, validation, or reporting. *Rationale: the orchestrator must stay responsive to operator commands and agent lifecycle events. Every time we let it do "just a little" analysis work inline, that work eventually grew until lifecycle handling starved.* **Functional requirements: conversational.** @@ -725,14 +728,16 @@ The substrate is specified as **behavior**, not mechanism. Whether it is a datab ### 9.3 Budget -- **FR-112**: The Orchestrator MUST track cumulative LLM spend (in currency) and cumulative wall-clock runtime across all runs of an evaluation, halting the fleet when either exceeds an operator-set cap. (US-5) -- **FR-113**: Spend tracking MUST account for every model call by every role. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates, and MUST surface what fraction of the reported total is estimated. +- **FR-112**: The Orchestrator MUST track cumulative settled LLM spend (in currency), outstanding cost reservations, and cumulative active wall-clock runtime across all runs of an evaluation. Active runtime MUST exclude operator-requested pauses and MUST NOT sum concurrent agent runtimes. The Orchestrator MUST invoke the FR-116a stop transition when settled spend reaches the spend cap or active runtime reaches the runtime cap. (US-5) +- **FR-112a**: Before dispatching any spend-creating model call, the budget governor MUST atomically reserve that call's conservative maximum cost against the spend cap, using the request's maximum output-token allowance and an immutable snapshot of the applicable provider/model rates. It MUST deny admission without contacting the provider when settled spend plus outstanding reservations plus the requested reservation would exceed the cap. On completion, failure, or cancellation it MUST settle the reservation exactly once against the billable actual or estimated cost and release any remainder. If a provider-reported charge exceeds its reservation, the governor MUST close new spend admission immediately and surface the discrepancy and any overrun as a degraded state. Budget admission enforces operator-authorized total spend; it is not the provider-rate pre-throttling prohibited by FR-105. +- **FR-113**: Spend tracking MUST record every model-call attempt by every role as a durable, idempotent accounting event. Each event MUST include a stable local call id; role and agent instance; provider and model; request, completion, or failure timestamps; outcome; input, output, cached-input, and reasoning token counts where reported; the immutable rate-card identity used for estimation; reserved cost; settled cost; and whether settled cost was provider-reported or estimated. A separately billed retry MUST have its own call id; replay of the same event MUST NOT count twice. Where the provider does not report cost directly, the system MUST estimate from token counts and configured rates. The surfaced estimated fraction MUST be estimated settled cost divided by total settled cost, not fraction of calls. - **FR-114**: Budget caps default to unset (unlimited). The pre-flight check (FR-010) SHOULD warn when both are unset. ### 9.4 Yield auto-stop -- **FR-115**: The system MUST compute trailing yield: confirmed findings, weighted by severity and by `exploited` status, divided by spend, over a trailing spend window. -- **FR-116**: The Orchestrator MUST halt the fleet when trailing yield falls below an operator-set threshold (US-4), but ONLY when all of: (a) at least one full trailing window of spend has accumulated; (b) a configured minimum runtime has elapsed; (c) the coverage-complete flag is set (FR-071). *Rationale: each precondition guards a real failure. Without (a) the metric is noise. Without (b) an early dry spell kills a young evaluation. Without (c) "we found nothing in the first six hours" stops the run before the required ground was covered.* +- **FR-115**: The system MUST compute trailing yield as confirmed findings, weighted by severity and by `exploited` status, divided by settled spend over a trailing spend window. For configured window size `W`, the window at time `t` MUST be the newest chronological suffix of settled accounting events whose total spend is at least `W` and whose total would be below `W` if its oldest event were removed. The window is not full while cumulative settled spend is below `W`; once full, its denominator is the suffix's actual settled spend, which may exceed `W` by the indivisible oldest call. Its numerator includes findings whose `true-positive` confirmation timestamp falls from that oldest event's settlement timestamp through `t`. Events with equal timestamps MUST be ordered by a durable sequence id so every consumer computes the same window. +- **FR-116**: The Orchestrator MUST halt the fleet when trailing yield falls below an operator-set threshold (US-4), but ONLY when all of: (a) at least one full trailing window of spend has accumulated; (b) a configured minimum active runtime has elapsed; (c) the coverage-complete flag is set (FR-071). *Rationale: each precondition guards a real failure. Without (a) the metric is noise. Without (b) an early dry spell kills a young evaluation. Without (c) "we found nothing in the first six hours" stops the run before the required ground was covered.* +- **FR-116a**: A hard-cap or low-yield stop decision MUST atomically close admission of new spend-creating calls before signaling shutdown, MUST allow already-reserved calls to settle, MAY perform bounded cleanup that creates no new LLM spend, and MUST durably record one stop reason and the settled and reserved budget state that caused it. Repeated evaluation of the same stop condition MUST be idempotent. - **FR-117**: Severity weights, the exploited multiplier, the trailing window size, the minimum runtime, and the threshold MUST be operator-configurable. The seed does not prescribe values. *Rationale: "at what return is this no longer worth the compute" is an organization-specific judgment, so the threshold is not prescribed. The shape of the severity weights is not arbitrary, though: the metric's job is to approximate economic value delivered per unit spend, so weights should track the relative real-world value of findings at each tier. A roughly geometric scale (constant ratio between adjacent tiers) does this and has two further properties a linear scale lacks: the trailing yield is dominated by the highest-severity finding in the window rather than by low-severity volume, and the threshold means the same thing across evaluations of very different targets. The seed authors used ~3.15× per tier (≈√10), calibrated against multi-year bug-bounty payout ratios, with a 2× multiplier for `exploited` findings to reflect the higher reviewer trust Constitution VII attaches to that flag. Builders SHOULD start geometric and tune the ratio to their own value data; builders SHOULD NOT default to a linear scale.* ### 9.5 Agent lifecycle limits @@ -768,11 +773,11 @@ Each subsection below defines the **interface contract** a Foundry implementatio ### 11.2 LLM provider -**Contract:** chat-completion with tool/function calling; system prompts; per-call token accounting; ideally prompt caching. Multiple model tiers (a strong model for investigation, a cheaper model for bulk classification) are SHOULD, not MUST. +**Contract:** chat-completion with tool/function calling; system prompts; a caller-set maximum output-token allowance; per-call accounting for billable token categories and direct cost where available; provider/model pricing that can be captured as an immutable rate-card snapshot; ideally prompt caching. Multiple model tiers (a strong model for investigation, a cheaper model for bulk classification) are SHOULD, not MUST. *Recommendation on tiering:* match model capability to task complexity rather than running one model everywhere. Roles whose output gates the pipeline's quality — Triager evidence reasoning (FR-051/FR-052), Validator reproduction (FR-060), Detector exploratory mode (FR-040), Cartographer authoring (§5.3) — benefit measurably from the strongest model you have access to. High-volume mechanical work — rule-sweep evaluation (FR-037), label/severity assignment, summarization — tolerates a cheaper tier with little quality loss. The seed does not name models; the heuristic is "the more open-ended the reasoning, the more the stronger model earns its cost". -[NEEDS CLARIFICATION: Which LLM provider(s) and model(s)? Will you use one model for all roles or tier by role? Is the provider API-compatible with a standard interface (OpenAI-style, Anthropic-style) or bespoke? Is prompt caching available? The seed authors used a hosted frontier-model API with prompt caching and two model tiers.] +[NEEDS CLARIFICATION: Which LLM provider(s) and model(s)? Will you use one model for all roles or tier by role? Is the provider API-compatible with a standard interface (OpenAI-style, Anthropic-style) or bespoke? Which billable token categories and direct-cost fields does it report, and where will immutable rate-card snapshots come from? Is prompt caching available? The seed authors used a hosted frontier-model API with prompt caching and two model tiers.] ### 11.3 Datastore @@ -841,12 +846,12 @@ A single evaluation is defined by a single configuration document with the follo | `detection` | Reference to the detection rule corpus (FR-041); per-evaluation rule overrides. | Detector | | `fleet` | Per-role: instance count, soft/hard session limits, role-specific prompt additions. | Orchestrator | | `sandbox` | Network allowlist (domains, CIDRs); read-only paths; extra mounts. | Orchestrator, isolation runtime | -| `budget` | Spend cap; time cap; yield threshold, window, minimum runtime; severity point weights. | Orchestrator, budget governor | +| `budget` | Spend cap; active-runtime cap; provider/model rate-card references; yield threshold, window, minimum runtime; severity point weights. | Orchestrator, budget governor | | `integrations` | Concrete bindings for each §11 surface: credentials references (not values), endpoints, model ids. | All | - **FR-126**: The configuration MUST be a single document (or a single directory treated as one) under version control alongside the evaluation's outputs. - **FR-127**: Secrets (API keys, tokens, testbed credentials) MUST NOT be stored in the configuration document. The configuration references them; a separate non-version-controlled mechanism supplies them. -- **FR-128**: The Orchestrator MUST hot-reload `budget` and `rules` changes at runtime, in addition to the `fleet` hot-reload required by FR-009. +- **FR-128**: The Orchestrator MUST hot-reload `budget` and `rules` changes at runtime, in addition to the `fleet` hot-reload required by FR-009. A spend cap lowered below settled spend plus outstanding reservations, or a runtime cap lowered below active runtime, MUST immediately invoke FR-116a. Raising the cap that caused an FR-011 restart block MUST clear that block. Hot reload and restart MUST NOT implicitly reset cumulative spend, reservations, runtime, or accounting history; reset requires an explicit operator action that is durably recorded. - **FR-128a**: Changes to `target`, `sandbox`, and `integrations` MAY require a restart. - **FR-129**: A configuration with unfilled required fields MUST fail FR-001 validation with a message naming each missing field. From 2845ac1b69d622872539148831b3764e251f2936 Mon Sep 17 00:00:00 2001 From: Omar Santos Date: Thu, 10 Sep 2026 14:33:48 -0400 Subject: [PATCH 2/2] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- constitution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/constitution.md b/constitution.md index a433051..5736b95 100644 --- a/constitution.md +++ b/constitution.md @@ -9,7 +9,7 @@ Downstream re-check : spec.md ✓ README.md ✓ principle-fr-matrix.md Follow-up TODOs : none Last sync : 2026-09-10 ═══════════════════════════════════════════════════════ -This block is regenerated on every constitution change; do not hand-edit below the rule. +This block is regenerated on every constitution change and on spec changes that require downstream re-validation; do not hand-edit below the rule. --> # Foundry Constitution