Repository navigation
Measure synthetic report stability under budgets - #415
Merged
Merged
Conversation
Report scoring had prompt-construction fixtures but no reusable frozen corpus for measuring repeat noise, evidence perturbations or phase assessability. Add seven authored cases and 84 tagged variants while keeping synthetic sensitivity separate from human rubric calibration. The corpus preserves clear engineering evidence when adding recognition gaps, distinguishes skipped phases from missing answers after a clear opportunity, and restricts identity swaps to transcript text. Pure metric checks pin null denominators and rubric bands. Identity sensitivity is a mean score shift rather than a spread difference, because the range of unequal samples differs from noise alone. The budget fold refuses further calls after unknown usage or exhausted observed ceilings. Three focused tests pass, and replacing score spread with zero fails the intended metric regression. No production prompt, report schema or provider request behavior changes.
The frozen report corpus needs a runner that measures the actual transport, semantic repairs, validation and salvage without losing failures or spending again after an interruption. Add an opt-in sequential evaluation path with a private observer on the single report transport path; normal report callers keep their policy and requests. Each call reserves its slot in an exclusive fsynced journal before sending and records decoded usage, failure stage and HTTP status. The journal pins the corpus, prompts, contract, model, seed, sources, executable and cumulative call and token ceilings. Resume skips finished rows and retries a row stopped by a 429, a 503 or a failure with reported usage, since those leave the ledger exact. In-flight work, other stops, torn records and changed identities refuse replay. Summaries count final outcomes, compare each item against its lowest complete repeat and name items left without one. Twenty-two credential-free tests cover resume, cancellation, timeout, usage gaps, budget boundaries, overflow, salvage, sidecar paths and each failure stage on the real request path; breaking each rule fails its test, and the full local gate passes. The credentialed pilot has not been run, so no provider stability or calibration result is claimed.
jserv
force-pushed
the
feat/report-evidence-evaluation
branch
from
October 21, 2026 16:54
7dfadc4 to
7f17000
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Report scoring had no frozen corpus for measuring how much a report moves under repeats and controlled edits. This adds seven synthetic cases with 84 variants (exact repeats, equivalent wording and code, semantic changes, transcript-only identity swaps, ASR gaps and phase assessability), the metrics that read them (score spread, rubric-band agreement, verdict flips, assessability agreement, and the identity mean shift against repeat noise), and a sequential runner for the production report transport under explicit call and token ceilings. The runner keeps a durable journal of every call, resumes without re-spending finished rows, retries a row stopped by a 429, a 503 or a failure with reported usage, and refuses every other stop. Normal report callers take the same transport path with no observer, so their requests and retry policy are unchanged.
The result is a synthetic sensitivity probe, not calibration:
docs/rubric-calibration.mdkeeps its NOT CALIBRATED status, and the probe is listed beside the other credentialed probes indocs/provider-cost-and-degradation.md.Verified with
./scripts/test.sh(exit 0, no skipped checks) andcargo test --lib report_eval(21 passed, 1 ignored). Each commit builds on its own. Breaking each resume, baseline, sidecar and metric rule fails its test. The credentialed pilot (report_eval_run, ignored) has not been run, so no provider stability result is claimed;tests/fixtures/report-eval/README.mdhas the bounded two-repeat pilot command.Summary by cubic
Report scoring had no frozen corpus for measuring how much a report moves under repeats and controlled edits. This adds seven synthetic cases with 84 variants, the metrics that read them, and a budget-bounded runner that executes the probe against the production report transport.
Written for commit 7f17000. Summary will update on new commits.