You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8c97fc3
Browse filesBrowse the repository at this point in the historyBrowse files
Add outcome adapters, the curated gold set, judges, statistics and Tier 2 forensics
Meta-improvement engine P2 (design §6, §11.2). Tier 2 is outcome-anchored:
a workflow that declares an outcome adapter gets a deterministic coverage
matrix and stage-of-loss attribution; a workflow without one keeps Tier 1
only, and nothing here makes a quality claim without an outcome record.
improve/adapters: the contract keeps two data models apart — the gold
matrix (match(): hit/miss/disputed/unlabeled per curated gold entry, the
carrier of recall and of stage-of-loss) and finding-level validity
(findings(): valid/invalid/unlabeled per produced finding, the carrier of
precision) — and keeps a paired judge's review-level scores as they are
(review_scores()). Every method reads outcome records already in the
store, so matrices and scores recompute identically; scores_from derives
the aggregates and stamps each with its source.
improve/gold.py: the curated gold set the raw inline comments lack.
gold_id = sha256(item + path + normalized concern)[:12]; `improve gold
draft` writes a draft from the raw comments for a human to curate, only a
file marked curated is ever used, a reworded concern is a new entry and
the file's sha is the version an experiment freezes.
improve/judges.py: run_judge — API judges through LLM.create, CLI judges
(cursor/claude/codex) through governed_subprocess in a fresh empty
workspace, tool-less by construction (a tool call in the stream discards
the verdict), both recorded as model_call records, both gated by the P3
governor when one is given.
improve/adapters/review_eval.py: the eval PR-review adapter — imports
judge_val verdicts (roles + blinding resolved) as judge_verdict outcomes,
runs the gold_match judge per gold entry (three votes, majority, a hit
needs a verbatim quote found in the review) into gold_match outcomes,
match() reads only those, findings are unlabeled (the judge scores whole
reviews). improve/adapters/rb_review.py: the bot's production adapter —
proxy labels from PR review threads (resolved/outdated/affirmative reply →
accepted, push-back → disputed, else silent), descriptive-only, no gold,
an unavailable thread is an "unavailable" record never a fabricated label.
improve/forensics.py: coverage_matrix from match(); the fixed S0–S10
taxonomy; forensics agents that see one unit's trace through read-only
tools (trace_get/trace_blob/trace_query, every result fenced as untrusted
data, no builtin tools), must cite record ids of that unit (an uncited or
unparseable answer is S0), run per miss cell by two model families with a
disagreement recorded as disputed; punch_list ranks loss per
(workflow, stage) with S10 routed to measurement health.
improve/stats.py: paired item-clustered t intervals (ported from
paired_analysis.py), items_required, the five labels, Cohen's kappa.
improve/meta.py + eval/dataset/meta/: the frozen meta-benchmark (cases
with human stage labels, lint samples), export/load, `improve meta
lint-check`. improve.forensics step joins the playbook; settings
improve_gt_dir/improve_judgments_dir/improve_eval_arm.
Review round 1 (gpt-6.1-sol-high) fixes: the forensics step collects
outcomes before the matrix (imports the paired judge's verdicts, runs
gold_match with the judge from settings.improve_judge, reports what it
collected and how many cells stay unlabeled); a lower-is-better metric is
judged on the mirrored interval; CLI judge blindness inspects message
content blocks and codex item types, not only event types; the codex judge
uses --skip-git-repo-check, the workspace, stdin for the prompt and the
nested item.text; trace_blob serves only blobs the unit's own records
reference.
Review round 2 fixes: the executor writes every unit's terminal
`step_result` decision with the rendered artifact (review_text /
answer_draft as blobs) and the published findings — the real review step
publishes them through step outputs and never wrote a decision, so the
adapter judged an empty body; gold_match now judges nothing when a unit has
no rendered review (cells stay unlabeled); the forensics step fetches
outcomes for every unit and derives finding-level scores independently of
gold coverage, so the bot adapter's proxy labels are collected; every
consumer of the declarations (executor, shadow hardening, cycle, forensics)
goes through enroll.declarations_for(settings), which honours
improve_workflows_dirs.
Review round 3 fixes: a curated gold file applies only to its own item
(repository and PR, and the head when it names one — PR numbers are
repository-local); a bot finding is labelled on ITS OWN thread (bot-authored,
carrying the finding's id or title, near its line) so a human's resolved
thread at the same line never labels it; gold_match records a verdict only
on a real majority of all three votes and leaves inconclusive cells
unrecorded so the next run retries them instead of caching a coin flip.
Round 4: id markers match whole (f-1 is not f-10), a lone nearby thread
with a different identity is not the finding's, and head-specific gold
requires that exact head; the title fallback matches the thread's own title exactly (its first
line minus the id marker), never a thread whose title extends the requested
one or carries another finding's marker, and rejects an ambiguous match;
a finding without a title matches by its full comment against the thread
body in the bot's publication format: the finding's own rendering
(comment plus its evidence field, as published) must equal the thread's
text minus marker and severity, so a comment that itself ends in an
Evidence line is distinct from one whose evidence was appended; findings
may say `file` or `path`.
Tests: test_improve_p2.py — gold draft/load/id discipline, judge paths
(governor reserve/settle, tool-using judge discarded, API path), paired
stats/labels/power/kappa, the eval adapter end to end with a fake judge
(idempotent imports, quote check, derived scores with sources), the bot
adapter (classification, proxy labels, cached outcomes, unavailable
threads), forensics (matrix, fenced tools, dispute, S0 on uncited claims,
punch list, measurement health), meta export/load/lint samples, the
forensics step's skips. Full suite green except test_mcp.py (host mcp 2.x).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
0 commit comments