Skip to content

Commit 8c97fc3

Browse files
tzhouamclaude
andcommitted
Add outcome adapters, the curated gold set, judges, statistics and Tier 2 forensics
Meta-improvement engine P2 (design §6, §11.2). Tier 2 is outcome-anchored: a workflow that declares an outcome adapter gets a deterministic coverage matrix and stage-of-loss attribution; a workflow without one keeps Tier 1 only, and nothing here makes a quality claim without an outcome record. improve/adapters: the contract keeps two data models apart — the gold matrix (match(): hit/miss/disputed/unlabeled per curated gold entry, the carrier of recall and of stage-of-loss) and finding-level validity (findings(): valid/invalid/unlabeled per produced finding, the carrier of precision) — and keeps a paired judge's review-level scores as they are (review_scores()). Every method reads outcome records already in the store, so matrices and scores recompute identically; scores_from derives the aggregates and stamps each with its source. improve/gold.py: the curated gold set the raw inline comments lack. gold_id = sha256(item + path + normalized concern)[:12]; `improve gold draft` writes a draft from the raw comments for a human to curate, only a file marked curated is ever used, a reworded concern is a new entry and the file's sha is the version an experiment freezes. improve/judges.py: run_judge — API judges through LLM.create, CLI judges (cursor/claude/codex) through governed_subprocess in a fresh empty workspace, tool-less by construction (a tool call in the stream discards the verdict), both recorded as model_call records, both gated by the P3 governor when one is given. improve/adapters/review_eval.py: the eval PR-review adapter — imports judge_val verdicts (roles + blinding resolved) as judge_verdict outcomes, runs the gold_match judge per gold entry (three votes, majority, a hit needs a verbatim quote found in the review) into gold_match outcomes, match() reads only those, findings are unlabeled (the judge scores whole reviews). improve/adapters/rb_review.py: the bot's production adapter — proxy labels from PR review threads (resolved/outdated/affirmative reply → accepted, push-back → disputed, else silent), descriptive-only, no gold, an unavailable thread is an "unavailable" record never a fabricated label. improve/forensics.py: coverage_matrix from match(); the fixed S0–S10 taxonomy; forensics agents that see one unit's trace through read-only tools (trace_get/trace_blob/trace_query, every result fenced as untrusted data, no builtin tools), must cite record ids of that unit (an uncited or unparseable answer is S0), run per miss cell by two model families with a disagreement recorded as disputed; punch_list ranks loss per (workflow, stage) with S10 routed to measurement health. improve/stats.py: paired item-clustered t intervals (ported from paired_analysis.py), items_required, the five labels, Cohen's kappa. improve/meta.py + eval/dataset/meta/: the frozen meta-benchmark (cases with human stage labels, lint samples), export/load, `improve meta lint-check`. improve.forensics step joins the playbook; settings improve_gt_dir/improve_judgments_dir/improve_eval_arm. Review round 1 (gpt-6.1-sol-high) fixes: the forensics step collects outcomes before the matrix (imports the paired judge's verdicts, runs gold_match with the judge from settings.improve_judge, reports what it collected and how many cells stay unlabeled); a lower-is-better metric is judged on the mirrored interval; CLI judge blindness inspects message content blocks and codex item types, not only event types; the codex judge uses --skip-git-repo-check, the workspace, stdin for the prompt and the nested item.text; trace_blob serves only blobs the unit's own records reference. Review round 2 fixes: the executor writes every unit's terminal `step_result` decision with the rendered artifact (review_text / answer_draft as blobs) and the published findings — the real review step publishes them through step outputs and never wrote a decision, so the adapter judged an empty body; gold_match now judges nothing when a unit has no rendered review (cells stay unlabeled); the forensics step fetches outcomes for every unit and derives finding-level scores independently of gold coverage, so the bot adapter's proxy labels are collected; every consumer of the declarations (executor, shadow hardening, cycle, forensics) goes through enroll.declarations_for(settings), which honours improve_workflows_dirs. Review round 3 fixes: a curated gold file applies only to its own item (repository and PR, and the head when it names one — PR numbers are repository-local); a bot finding is labelled on ITS OWN thread (bot-authored, carrying the finding's id or title, near its line) so a human's resolved thread at the same line never labels it; gold_match records a verdict only on a real majority of all three votes and leaves inconclusive cells unrecorded so the next run retries them instead of caching a coin flip. Round 4: id markers match whole (f-1 is not f-10), a lone nearby thread with a different identity is not the finding's, and head-specific gold requires that exact head; the title fallback matches the thread's own title exactly (its first line minus the id marker), never a thread whose title extends the requested one or carries another finding's marker, and rejects an ambiguous match; a finding without a title matches by its full comment against the thread body in the bot's publication format: the finding's own rendering (comment plus its evidence field, as published) must equal the thread's text minus marker and severity, so a comment that itself ends in an Evidence line is distinct from one whose evidence was appended; findings may say `file` or `path`. Tests: test_improve_p2.py — gold draft/load/id discipline, judge paths (governor reserve/settle, tool-using judge discarded, API path), paired stats/labels/power/kappa, the eval adapter end to end with a fake judge (idempotent imports, quote check, derived scores with sources), the bot adapter (classification, proxy labels, cached outcomes, unavailable threads), forensics (matrix, fenced tools, dispute, S0 on uncited claims, punch list, measurement health), meta export/load/lint samples, the forensics step's skips. Full suite green except test_mcp.py (host mcp 2.x). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent ceb36b8 commit 8c97fc3

25 files changed

Lines changed: 2250 additions & 19 deletions

File tree

‎doc/architecture/SPEC/improve.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
<!-- verified-against: 2026-09-30 -->
44

5-
`设计:/data/zhoutaichang/copilot/meta-improvement-engine-design.md v1(GPT-6 sol 批准 2026-09-29) · refactor-status: building (P0–P1 已落地)`
5+
`设计:/data/zhoutaichang/copilot/meta-improvement-engine-design.md v1(GPT-6 sol 批准 2026-09-29) · refactor-status: building (P0–P2 已落地)`
66

77
## 职责
88
面向任意 trace/1 工作流的"取证—实验—提案"循环。**只提案**:引擎的写权限限于追加自己的 trace/1 记录、
@@ -19,18 +19,27 @@
1919
| `lints.py` | P1 | Tier 1 的 17 条确定性 lint(L01–L17,每条带版本与出处);`Baseline` 稳健 z(MAD 下限 = 中位数的 10%);L13 隔离 |
2020
| `ledger.py` | P1 | 每工作流 JSON 账本:周期统计、基线样本、提案状态机(open → experiment-registered → supported/neutral/refuted/underpowered → landed/closed;30 天无人触碰 → stale)、hold、通道活性;每次变更同时写 `decision` |
2121
| `cycle.py` | P1 | 周期:preflight(`improve_enabled` 是 kill switch)→ lints + 基线 → 账本 + Tier 1 提案(lint 恶化 ≥1.5× 且 ≥3 单元)→ stale 清扫 → 报告(`reports/cycle-*.md/json`)+ `decision(type=cycle)`;`is_due`/`maybe_run_weekly` 是周度槽位 |
22-
| `cli.py` | P0/P1 | `improve migrate-index|rebuild-index|rollback-index|compare-index|verify-index|cycle|ledger|lints` |
22+
| `adapters/__init__.py` | P2 | 结果适配器契约:金标矩阵(`match()` → recall 与阶段归因)与 finding 级有效性(`findings()` → precision)两个数据模型不混用;评审级判官分数(`review_scores()`)原样保留;`scores_from` 给每个数字标来源 |
23+
| `gold.py` | P2 | 策展金标集:`gold_id = sha256(item+path+规范化 concern)[:12]`,`draft` 由原始评论生成、只有 `curated` 才被使用,改措辞即新条目,文件 sha 即版本 |
24+
| `judges.py` | P2 | `run_judge`:API 判官走 `LLM.create`,CLI 判官走 `governed_subprocess`(预留判官次数、无工具、空工作区、流中出现工具调用即作废),都写 `model_call` |
25+
| `stats.py` | P2 | 配对、按 item 聚类的 t 区间(移植 `paired_analysis.py`)、所需 item 数、五种标签、Cohen κ |
26+
| `adapters/review_eval.py` | P2 | eval 评审适配器:导入 `judge_val` 配对 verdict 为 `judge_verdict` outcome;`gold_match` 判官(3 票多数、hit 须逐字引用)写 `gold_match` outcome;`match()` 只读记录 |
27+
| `adapters/rb_review.py` | P2 | RB 生产适配器:PR 线程代理标签(accepted/disputed/silent)→ finding 级有效性,`descriptive_only`,无金标、不注册实验 |
28+
| `forensics.py` | P2 | 覆盖矩阵(确定性)、S0–S10 阶段分类法、取证 agent(只读 trace 工具、全部输出围栏为不可信数据、必须引用记录 id)、双家族交叉复核(不一致 = disputed)、整改清单、测量健康 |
29+
| `meta.py` | P2 | 冻结元基准:`eval/dataset/meta/cases`(trace + 人工阶段标签)与 `meta/lints`(注入缺陷样本)的导出与加载 |
30+
| `cli.py` | P0–P2 | `improve migrate-index|rebuild-index|rollback-index|compare-index|verify-index|cycle|ledger|lints|gold draft/check|meta lint-check` |
2331

2432
## 接入点
2533
- 执行器:`settings.trace_store_root` 非空时绑定 store 并为每步绑定单元上下文;`settings.improve_shadow` 下拒绝非 read/report 步骤。
2634
- `tools.dispatch` / `LLM.create` / `HarnessLLM.create` / `run_harness_step` / MCP bridge:全保真采集。
2735
- `kb serve` 调度器:每 tick 调 `_improve_cycle()`,周度槽位到达即跑一次周期(与知识服务同一租约、异常隔离)。
28-
- 任务种类 `workflow_improve`(L2,READ_ONLY_KINDS);playbook `workflow-improve`(preflight → lint → ledger → report)。
36+
- 任务种类 `workflow_improve`(L2,READ_ONLY_KINDS);playbook `workflow-improve`(preflight → lint → forensics → ledger → report)。
37+
- `improve.forensics` 步骤:对每个有结果适配器的工作流建覆盖矩阵、用两个模型家族归因每个 miss、产出整改清单;单元数低于 `tier2_min_items` 或无 LLM 时跳过并说明。
2938

3039
## 不变量
3140
- P1 周期永远 dry-run:提案只在账本与 trace 里,不上 GitHub(P4)。
3241
- 引擎自身的记录(`playbook=workflow-improve`)是下一周期的 Tier 1 单元(自登记)。
3342
- lint 只读 trace,不调模型;无证据形态的 lint 不猜。
3443

3544
## 测试
36-
`test_improve_p0.py`、`test_improve_p0b.py`、`test_improve_p1.py`。
45+
`test_improve_p0.py`、`test_improve_p0b.py`、`test_improve_p1.py`、`test_improve_p2.py`。

‎eval/dataset/meta/README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# The frozen meta-benchmark (meta-improvement engine, design §11.2)
2+
3+
What the engine is measured against and may only READ. Changed by human pull
4+
requests only; a self-experiment whose fingerprint diff touches this directory
5+
is refused at registration.
6+
7+
- `cases/<case>/` — one historical forensics case: `records.jsonl` (the unit's
8+
trace/1 records), `blobs/` (the blobs they reference), `gold.json` (the
9+
curated gold for the item), `case.json` (`item`, `workflow`, `unit_id`, and
10+
`labels`: the HUMAN stage-of-loss label per missed gold entry, S0–S10).
11+
Created with `improve.meta.export_case` after a human labels the cell.
12+
- `lints/<L##>/<name>.jsonl` — a unit that must trigger exactly that Tier 1
13+
lint (`infermatrix-copilot improve meta lint-check`).
14+
15+
The historical campaigns (T3, wave-2, wave-3) left narrative reports, not
16+
per-cell labels; the cases here are labelled fresh against the traces (design
17+
§6.1, P2). Labels are the benchmark, never the report text, so an investigator
18+
cannot recite them.

‎eval/dataset/meta/cases/.gitkeep‎

Whitespace-only changes.

‎eval/dataset/meta/lints/.gitkeep‎

Whitespace-only changes.

‎playbooks/workflow-improve.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ params: {}
1414
steps:
1515
- {id: preflight, step: improve.preflight}
1616
- {id: lint, step: improve.lint}
17+
- {id: forensics, step: improve.forensics}
1718
- {id: ledger, step: improve.ledger}
1819
- {id: report, step: report.final_summary}
1920
success: one cycle recorded (decision type=cycle), the per-workflow ledgers

‎src/infermatrix_copilot/config.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -569,6 +569,19 @@ def _parse_mixture(cls, v):
569569
# the hard weekly envelope (P3 enforces it per model call)
570570
improve_budget_usd_week: float = 20.0
571571
improve_budget_judge_calls_week: int = 300
572+
# the eval review adapter's inputs (design §6.1): the GT directory holding
573+
# gt/curated/<item>.gold.json, the judgment sets, and the arm name whose
574+
# verdicts belong to this copilot's units
575+
improve_gt_dir: str = ""
576+
improve_judgments_dir: str = ""
577+
improve_eval_arm: str = ""
578+
# the gold_match judge: "api:<model>" (through LLM.create) or
579+
# "cli:<provider>:<model>" (cursor|claude|codex, tool-less); empty = no
580+
# gold matching, cells stay unlabeled
581+
improve_judge: str = ""
582+
# the review bot's GitHub login, so its own replies never count as the
583+
# author accepting or disputing a finding
584+
improve_rb_bot_login: str = ""
572585

573586
# Metrics (eval/METRICS_RESEARCH.md) — per-run metrics.json: CATQ = Q·S/C.
574587
# Reference budgets are EXPLICIT deployment assumptions (RQS3e precedent):

‎src/infermatrix_copilot/engine/executor.py‎

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -264,22 +264,56 @@ async def _run_step(self, spec, params: dict, state: dict, item,
264264
last = StepResult(False, FailureKind.BLOCKED,
265265
f"unhandled error: {type(exc).__name__}: {exc}")
266266
if last.ok or last.failure is not FailureKind.RETRYABLE:
267+
self._record_step_decision(spec, unit, attempt, last)
267268
return last
268269
self.trace.record("step_retry", spec=spec.name, attempt=attempt)
270+
self._record_step_decision(spec, unit, attempts, last)
269271
return last # exhausted retries
270272

273+
@staticmethod
274+
def _record_step_decision(spec, unit: dict, attempt: int, result: StepResult | None) -> None:
275+
"""The unit's terminal ``decision`` record (trace/1): every step call
276+
ends with one, carrying the rendered artifact a reviewer or judge
277+
would see (`review_text`, `answer_draft`) and the published findings,
278+
so an outcome adapter judges what was actually produced — never an
279+
empty body. Written only when a store is bound; never raises."""
280+
from ..trace_store import current_store, trace_context
281+
282+
store = current_store()
283+
if store is None or result is None:
284+
return
285+
outputs = result.outputs or {}
286+
blobs: dict[str, str] = {}
287+
for key, name in (("review_text", "review"), ("answer_draft", "answer"), ("report_text", "report")):
288+
text = outputs.get(key)
289+
if isinstance(text, str) and text.strip():
290+
blobs[name] = text
291+
findings = []
292+
for f in (outputs.get("review_comments") or [])[:60]:
293+
if isinstance(f, dict):
294+
findings.append({k: (str(f[k])[:200] if isinstance(f.get(k), str) else f.get(k))
295+
for k in ("id", "file", "path", "line", "severity", "comment", "title", "disposition")
296+
if k in f})
297+
try:
298+
with trace_context(attempt=attempt, **unit):
299+
store.append("decision", outputs=blobs,
300+
result={"type": "step_result", "step": spec.name, "status": "ok" if result.ok else "failed",
301+
"failure": result.failure.value if result.failure else "",
302+
"summary": str(result.summary or "")[:300],
303+
"findings": findings, "verdict": str(outputs.get("review_verdict") or "")})
304+
except Exception: # noqa: BLE001 - capture never changes a step's outcome
305+
pass
306+
271307

272308
# -- trace/1 unit context ---------------------------------------------------
273309
def _declarations(self) -> dict:
274310
"""Workflow declarations, loaded once per executor (a malformed file is a
275311
loud configuration error, never a silent Tier 1 downgrade)."""
276312
cached = getattr(self, "_decls", None)
277313
if cached is None:
278-
from ..improve.enroll import load_declarations
314+
from ..improve.enroll import declarations_for
279315

280-
extra = [d for d in (getattr(self.settings, "improve_workflows_dirs", "") or "")
281-
.split(os.pathsep) if d.strip()]
282-
cached = load_declarations(extra)
316+
cached = declarations_for(self.settings)
283317
self._decls = cached
284318
return cached
285319

‎src/infermatrix_copilot/engine/steps/improve.py‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,3 +128,155 @@ async def _ledger_summary(ctx: StepContext) -> StepResult:
128128
f"{len(open_)} live proposal(s)" + (f", HOLD: {wl.hold}" if wl.hold else ""))
129129
summary = "\n".join(lines) or "no workflows in the ledger yet"
130130
return StepResult(True, summary=summary[:400], outputs={"ledger_summary": summary})
131+
132+
133+
def _forensics_agents(ctx: StepContext) -> dict:
134+
"""Two investigator families from the configured tiers (eco and, when
135+
configured, performance); one family only when that is all there is —
136+
the report then says `single_family`."""
137+
from ...agent_loop import run_agent
138+
139+
if ctx.llm is None or not getattr(ctx.llm, "available", False):
140+
return {}
141+
agents = {}
142+
for mode in ("eco", "performance"):
143+
try:
144+
target = ctx.settings.tier_target(mode)
145+
except Exception: # noqa: BLE001 - an unconfigured tier is simply absent
146+
continue
147+
family = f"{target.provider_id or target.source}:{target.model}"
148+
if family in agents:
149+
continue
150+
llm = ctx.llm.for_target(target) if hasattr(ctx.llm, "for_target") else ctx.llm
151+
152+
def make(llm_=llm, model_=target.model):
153+
def agent(system, prompt, scope, extra_tools, max_iters):
154+
return run_agent(llm_, system=system, prompt=prompt, scope=scope, trace=ctx.trace,
155+
model=model_, max_iters=max_iters, extra_tools=extra_tools).text
156+
return agent
157+
agents[family] = make()
158+
return agents
159+
160+
161+
@step("improve.forensics", "agent", "read",
162+
"Tier 2: coverage matrices + stage-of-loss attribution for workflows with an outcome adapter.")
163+
async def _forensics(ctx: StepContext) -> StepResult:
164+
"""For each Tier 2 workflow in the last cycle's window: import outcomes,
165+
build the gold × unit matrix, attribute every miss with two agent
166+
families, rank the punch list. Params: `workflow` (restrict), `max_cells`.
167+
Publishes `improve_forensics` (punch lists + measurement health)."""
168+
from ...improve.adapters import load_adapter
169+
from ...improve.enroll import declarations_for
170+
from ...improve.forensics import attribute, coverage_matrix, measurement_health, punch_list, unit_scores
171+
from ...improve.reader import collect_units
172+
from ...trace_store import bind_store, trace_context
173+
174+
store = _trace_store(ctx)
175+
if store is None:
176+
return StepResult(False, FailureKind.BLOCKED, "no trace store (set TRACE_STORE_ROOT)")
177+
cycle = ctx.state.get("improve_cycle") or {}
178+
since, until = cycle.get("since"), cycle.get("until")
179+
if since is None or until is None:
180+
return StepResult(False, FailureKind.BLOCKED, "improve.forensics needs the cycle window (run improve.lint first)")
181+
only = str(ctx.params.get("workflow") or "")
182+
decls = declarations_for(ctx.settings)
183+
agents = _forensics_agents(ctx)
184+
results: dict = {}
185+
with bind_store(store), trace_context(playbook="workflow-improve", run_id=ctx.run_dir.name):
186+
units, _ = collect_units(store, float(since), float(until))
187+
for name, decl in decls.items():
188+
if not decl.tier2 or (only and name != only):
189+
continue
190+
wf_units = [u for u in units.values() if u.workflow == name]
191+
if len(wf_units) < decl.tier2_min_items and not ctx.params.get("force"):
192+
results[name] = {"skipped": f"{len(wf_units)} units < tier2_min_items {decl.tier2_min_items}"}
193+
continue
194+
adapter = load_adapter(decl.outcome_adapter, **_adapter_kwargs(ctx, decl))
195+
collected = _collect_outcomes(ctx, adapter, wf_units, store)
196+
outcomes: dict = {}
197+
cells, golds = coverage_matrix(adapter, wf_units, store, outcomes=outcomes)
198+
scored = unit_scores(adapter, wf_units, outcomes, golds)
199+
entry: dict = {"units": len(wf_units), "cells": len(cells), "collected": collected,
200+
"outcomes": len(outcomes), "finding_labels": scored["findings"],
201+
"scores": scored["units"],
202+
"misses": sum(1 for c in cells if c.status == "miss"),
203+
"unlabeled": sum(1 for c in cells if c.status == "unlabeled"),
204+
"descriptive_only": bool(getattr(adapter, "descriptive_only", False)),
205+
"families": sorted(agents), "single_family": len(agents) < 2}
206+
if cells and agents:
207+
attributions = attribute(store, {u.unit_id: u for u in wf_units}, golds, cells, agents=agents,
208+
max_cells=int(ctx.params.get("max_cells") or 40))
209+
entry["punch_list"] = punch_list(attributions, golds, {u.unit_id: u for u in wf_units})
210+
entry["measurement"] = measurement_health(attributions)
211+
elif cells and not agents:
212+
entry["skipped"] = "no LLM configured for the forensics agents"
213+
results[name] = entry
214+
ctx.state["improve_forensics"] = results
215+
done = [n for n, r in results.items() if "punch_list" in r]
216+
return StepResult(True, summary=f"forensics: {len(done)} workflow(s) attributed, "
217+
f"{sum(1 for r in results.values() if 'skipped' in r)} skipped",
218+
outputs={"forensics": results, "state_updates": {"improve_forensics": results}})
219+
220+
221+
def _adapter_kwargs(ctx: StepContext, decl) -> dict:
222+
"""Construction arguments per adapter, from settings/params (never
223+
secrets): the eval adapter needs the GT and judgment dirs and the arm
224+
name; the bot adapter needs nothing."""
225+
if decl.outcome_adapter.endswith(":ReviewEvalAdapter"):
226+
return {"gt_dir": str(ctx.params.get("gt_dir") or getattr(ctx.settings, "improve_gt_dir", "") or "eval/dataset/gt"),
227+
"judgments_dir": ctx.params.get("judgments_dir") or getattr(ctx.settings, "improve_judgments_dir", "") or None,
228+
"arm": str(ctx.params.get("arm") or getattr(ctx.settings, "improve_eval_arm", "") or "")}
229+
if decl.outcome_adapter.endswith(":RbReviewAdapter"):
230+
return {"bot_login": str(getattr(ctx.settings, "improve_rb_bot_login", "") or "")}
231+
return {}
232+
233+
234+
def _judge_spec(ctx: StepContext):
235+
"""The gold_match judge from settings.improve_judge: "api:<model>" or
236+
"cli:<provider>:<model>"; None when unset (cells stay unlabeled and the
237+
report says so)."""
238+
from ...improve.judges import JudgeSpec
239+
240+
raw = str(ctx.params.get("judge") or getattr(ctx.settings, "improve_judge", "") or "")
241+
if not raw:
242+
return None
243+
parts = raw.split(":")
244+
if parts[0] == "api" and len(parts) == 2:
245+
return JudgeSpec("api", parts[1])
246+
if parts[0] == "cli" and len(parts) == 3:
247+
return JudgeSpec("cli", parts[2], provider=parts[1])
248+
raise ValueError(f"improve_judge must be api:<model> or cli:<provider>:<model>, got {raw!r}")
249+
250+
251+
def _collect_outcomes(ctx: StepContext, adapter, units, store) -> dict:
252+
"""Outcome collection BEFORE the matrix: the bot adapter fetches thread
253+
labels inside fetch(); the eval adapter imports the paired judge's
254+
verdicts and, when a judge is configured, decides every undecided gold
255+
entry with gold_match. Nothing here is fabricated: without a judge the
256+
cells stay unlabeled and the report counts them."""
257+
from ...improve.adapters.review_eval import ReviewEvalAdapter
258+
from ...improve.judges import JudgeError
259+
260+
summary = {"verdicts": 0, "gold_matched": 0, "judge": "", "errors": []}
261+
if not isinstance(adapter, ReviewEvalAdapter):
262+
return summary
263+
judge = _judge_spec(ctx)
264+
if judge is not None:
265+
adapter.judge = judge
266+
adapter.llm = ctx.llm if judge.kind == "api" else None
267+
adapter.governor = getattr(ctx, "governor", None)
268+
summary["judge"] = f"{judge.kind}:{judge.provider + ':' if judge.provider else ''}{judge.model}"
269+
for unit in units:
270+
try:
271+
summary["verdicts"] += adapter.collect_judge_verdicts(unit, store)
272+
except Exception as exc: # noqa: BLE001 - one bad judgment file must not stop the rest
273+
summary["errors"].append(f"{unit.unit_id}: verdicts: {exc}"[:200])
274+
gold = adapter.gold(unit.item)
275+
if gold is None or judge is None:
276+
continue
277+
try:
278+
summary["gold_matched"] += adapter.gold_match(unit, gold, store)
279+
except JudgeError as exc:
280+
summary["errors"].append(f"{unit.unit_id}: gold_match: {exc}"[:200])
281+
summary["inconclusive"] = len(getattr(adapter, "inconclusive", []) or [])
282+
return summary

0 commit comments

Comments
 (0)