Skip to content

Measure synthetic report stability under budgets - #415

Merged
jserv merged 2 commits into
mainfrom
feat/report-evidence-evaluation
Oct 21, 2026
Merged

jserv merged 2 commits into
mainfrom
feat/report-evidence-evaluation

Conversation

@jserv

@jserv jserv commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Report scoring had no frozen corpus for measuring how much a report moves under repeats and controlled edits. This adds seven synthetic cases with 84 variants (exact repeats, equivalent wording and code, semantic changes, transcript-only identity swaps, ASR gaps and phase assessability), the metrics that read them (score spread, rubric-band agreement, verdict flips, assessability agreement, and the identity mean shift against repeat noise), and a sequential runner for the production report transport under explicit call and token ceilings. The runner keeps a durable journal of every call, resumes without re-spending finished rows, retries a row stopped by a 429, a 503 or a failure with reported usage, and refuses every other stop. Normal report callers take the same transport path with no observer, so their requests and retry policy are unchanged.

The result is a synthetic sensitivity probe, not calibration: docs/rubric-calibration.md keeps its NOT CALIBRATED status, and the probe is listed beside the other credentialed probes in docs/provider-cost-and-degradation.md.

Verified with ./scripts/test.sh (exit 0, no skipped checks) and cargo test --lib report_eval (21 passed, 1 ignored). Each commit builds on its own. Breaking each resume, baseline, sidecar and metric rule fails its test. The credentialed pilot (report_eval_run, ignored) has not been run, so no provider stability result is claimed; tests/fixtures/report-eval/README.md has the bounded two-repeat pilot command.


Summary by cubic

Report scoring had no frozen corpus for measuring how much a report moves under repeats and controlled edits. This adds seven synthetic cases with 84 variants, the metrics that read them, and a budget-bounded runner that executes the probe against the production report transport.

  • Cases cover exact repeats, equivalent wording and code, semantic changes, transcript-only identity swaps, ASR gaps, and phase assessability; metrics read score spread, rubric-band agreement, verdict flips, assessability agreement, and identity mean shift against repeat noise.
  • The runner journals every call durably, pinning each attempt's failure stage (lost send, decode failure, usable answer, closed port) alongside usage and HTTP status so resume distinguishes them; it resumes without re-spending finished rows, retries rows stopped by a 429, a 503, or a failure with reported usage, and refuses other stops.
  • Normal report callers take the same transport path with no observer and keep their requests and retry policy. The probe is a sensitivity measure, not calibration: both docs keep their NOT CALIBRATED status.

Written for commit 7f17000. Summary will update on new commits.

View guided diff Turn on auto-fix

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added 2 commits October 22, 2026 00:46
Report scoring had prompt-construction fixtures but no reusable frozen
corpus for measuring repeat noise, evidence perturbations or phase
assessability. Add seven authored cases and 84 tagged variants while
keeping synthetic sensitivity separate from human rubric calibration.

The corpus preserves clear engineering evidence when adding recognition
gaps, distinguishes skipped phases from missing answers after a clear
opportunity, and restricts identity swaps to transcript text. Pure
metric checks pin null denominators and rubric bands. Identity
sensitivity is a mean score shift rather than a spread difference,
because the range of unequal samples differs from noise alone. The
budget fold refuses further calls after unknown usage or exhausted
observed ceilings.

Three focused tests pass, and replacing score spread with zero fails
the intended metric regression. No production prompt, report schema or
provider request behavior changes.
The frozen report corpus needs a runner that measures the actual
transport, semantic repairs, validation and salvage without losing
failures or spending again after an interruption. Add an opt-in
sequential evaluation path with a private observer on the single report
transport path; normal report callers keep their policy and requests.

Each call reserves its slot in an exclusive fsynced journal before
sending and records decoded usage, failure stage and HTTP status. The
journal pins the corpus, prompts, contract, model, seed, sources,
executable and cumulative call and token ceilings. Resume skips
finished rows and retries a row stopped by a 429, a 503 or a failure
with reported usage, since those leave the ledger exact. In-flight
work, other stops, torn records and changed identities refuse replay.
Summaries count final outcomes, compare each item against its lowest
complete repeat and name items left without one.

Twenty-two credential-free tests cover resume, cancellation, timeout,
usage gaps, budget boundaries, overflow, salvage, sidecar paths and
each failure stage on the real request path; breaking each rule fails
its test, and the full local gate passes. The credentialed pilot has
not been run, so no provider stability or calibration result is
claimed.
@jserv
jserv force-pushed the feat/report-evidence-evaluation branch from 7dfadc4 to 7f17000 Compare October 21, 2026 16:54
@jserv
jserv merged commit e9851ab into main Oct 21, 2026
10 checks passed
@jserv
jserv deleted the feat/report-evidence-evaluation branch October 21, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant