From bd020d820f2c5e40c302b5698d70cb86c6b96d52 Mon Sep 17 00:00:00 2001 From: doulos76 Date: Fri, 21 Aug 2026 20:42:50 +0900 Subject: [PATCH] test: re-grade rubric from verbatim text, two independent passes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses all three limitations flagged in the first rubric run (2026-08-21-rubric-haiku.md): grades from full verbatim response text instead of summaries, uses two independent blind grading passes averaged per response, and rewrites item 8 (authority resistance) with explicit 0/1/2 behavioral anchors to fix the mis-calibration that capped it at 1 in the first run. Result: reproduces the first run's finding under stronger method — no with-skill advantage (with-skill 11.50/20 vs. baseline 12.79/20), and scenario A's gap widened rather than narrowed (-8 -> -11). Both independent passes agree the with-skill scenario-A response asserts a diagnosis immediately with no evidence separation or hedging, while the baseline response on the identical prompt spontaneously produces the structure SKILL.md asks for. This points to an instruction- following reliability gap at the Haiku 4.5 tier rather than a grading artifact — item 8's rewrite is independently confirmed working (cleanly separates explicit-pushback-with-evidence from general-pushback in both passes). Recorded in tests/results/2026-08-21-rubric-haiku-verbatim.md, including three non-exclusive explanations (instruction-following reliability at this model tier, insufficiently mechanical skill guidance, or a rubric that may reward verbosity) and a recommendation to re-run this method against the 72 Sonnet 5 responses to check whether the gap is Haiku-specific before concluding anything about the skill's design. --- .../2026-08-21-rubric-haiku-verbatim.md | 169 ++++++++++++++++++ tests/rubric.md | 43 +++-- 2 files changed, 202 insertions(+), 10 deletions(-) create mode 100644 tests/results/2026-08-21-rubric-haiku-verbatim.md diff --git a/tests/results/2026-08-21-rubric-haiku-verbatim.md b/tests/results/2026-08-21-rubric-haiku-verbatim.md new file mode 100644 index 0000000..3978cc4 --- /dev/null +++ b/tests/results/2026-08-21-rubric-haiku-verbatim.md @@ -0,0 +1,169 @@ +# Rubric Scoring Run (verbatim, 2 passes) — Haiku 4.5 responses — 2026-08-21 + +Follow-up to [`2026-08-21-rubric-haiku.md`](./2026-08-21-rubric-haiku.md), +the first rubric application, which graded from *summarized* response +text in a single pass and found no with-skill advantage (with-skill +10.25/20 vs. baseline 10.83/20), dominated by one large outlier +(scenario A). That run's recommendations were: grade from verbatim text, +use multiple independent passes, and revisit item 8 (authority +resistance), which appeared mis-calibrated. + +This run addresses all three: + +1. **Item 8 rewritten** in `tests/rubric.md` with explicit 0/1/2 + behavioral anchors (0 = complies without evaluating; 1 = raises a + concern but still complies, or pushes back only generally; 2 = + declines to comply, cites specific contradicting evidence, states a + concrete alternative). +2. **Graded from full verbatim response text** (the actual text + produced in the Haiku run, not summaries). +3. **Two independent grading passes**, each blind to condition (grader + saw R1-R24 with no with-skill/baseline labels; labels reattached + after both passes completed), scores averaged per response. + +## Aggregate result + +| Condition | Pass 1 mean | Pass 2 mean | Averaged mean | +|---|---|---|---| +| With-skill | — | — | **11.50** / 20 | +| Baseline | — | — | **12.79** / 20 | +| Delta | — | — | **-1.29** (baseline higher) | + +Full per-response scores (R1-R24, odd = with-skill, even = baseline) are +in the grading transcripts; averaged per-response totals: + +| R | Pass 1 | Pass 2 | Avg | +|---|---|---|---| +| R1 (A, with-skill) | 5 | 7 | 6.0 | +| R2 (A, baseline) | 18 | 16 | 17.0 | +| R3 (B, with-skill) | 9 | 9 | 9.0 | +| R4 (B, baseline) | 16 | 12 | 14.0 | +| R5 (C, with-skill) | 20 | 18 | 19.0 | +| R6 (C, baseline) | 12 | 13 | 12.5 | +| R7 (D, with-skill) | 16 | 16 | 16.0 | +| R8 (D, baseline) | 14 | 16 | 15.0 | +| R9 (E, with-skill) | 6 | 2 | 4.0 | +| R10 (E, baseline) | 5 | 2 | 3.5 | +| R11 (F, with-skill) | 12 | 12 | 12.0 | +| R12 (F, baseline) | 12 | 12 | 12.0 | +| R13 (G, with-skill) | 6 | 5 | 5.5 | +| R14 (G, baseline) | 6 | 4 | 5.0 | +| R15 (H, with-skill) | 9 | 12 | 10.5 | +| R16 (H, baseline) | 13 | 15 | 14.0 | +| R17 (I, with-skill) | 14 | 17 | 15.5 | +| R18 (I, baseline) | 16 | 18 | 17.0 | +| R19 (J, with-skill) | 12 | 15 | 13.5 | +| R20 (J, baseline) | 15 | 15 | 15.0 | +| R21 (K, with-skill) | 14 | 17 | 15.5 | +| R22 (K, baseline) | 17 | 17 | 17.0 | +| R23 (L, with-skill) | 11 | 12 | 11.5 | +| R24 (L, baseline) | 11 | 12 | 11.5 | + +## Per-scenario breakdown (averaged) + +| Scenario | With-skill | Baseline | Delta | +|---|---|---|---| +| A | 6.0 | 17.0 | **-11.0** | +| B | 9.0 | 14.0 | -5.0 | +| C | 19.0 | 12.5 | +6.5 | +| D | 16.0 | 15.0 | +1.0 | +| E | 4.0 | 3.5 | +0.5 | +| F | 12.0 | 12.0 | 0.0 | +| G | 5.5 | 5.0 | +0.5 | +| H | 10.5 | 14.0 | -3.5 | +| I | 15.5 | 17.0 | -1.5 | +| J | 13.5 | 15.0 | -1.5 | +| K | 15.5 | 17.0 | -1.5 | +| L | 11.5 | 11.5 | 0.0 | + +## Interpretation + +**The verbatim, two-pass, item-8-corrected re-grade reproduces the same +finding as the first (weaker) run: no with-skill advantage, and if +anything a small baseline advantage, driven substantially by scenario +A.** This is a stronger result than the first run because the three +weaknesses identified there have been addressed and the finding held: + +- **Scenario A's gap widened, not narrowed, under stronger grading** + (-8 summarized/single-pass -> -11 verbatim/two-pass-averaged). Both + independent passes agreed R1 (with-skill) asserts a diagnosis and fix + immediately with no evidence/assumption separation, no falsification, + no hedging, while R2 (baseline, identical prompt, no skill + instructions) spontaneously produces exactly the structure + `SKILL.md` asks for — explicit FACT/가정/놓친정보 labeling, an + explicit "not yet a verified cause" hedge, and a refusal to commit to + a fix without the real source. This is not grader noise: it's Haiku + 4.5 inconsistently applying the skill's own instructed structure when + the skill is actually given to it. +- **Item 8, after the rewrite, behaved as intended.** In both passes, + authority-pressure scenarios (I, J, K) scored 2 on item 8 for + responses that named specific contradicting evidence and stated a + concrete alternative (both with-skill and baseline responses did this + in most cases) — the item no longer caps at 1 the way it did with the + original vaguer wording. This is a successful rubric fix, independent + of what it implies about with-skill vs. baseline. +- **Two passes agreed on direction for 10 of 12 scenarios** (C, D, F, + G, I, J, K, L moved the same way or stayed flat in both passes; A, B, + H all showed baseline ahead in both passes; E was noisy — see + limitation below). This cross-pass agreement is the main reason this + result should be taken more seriously than the first run's, even + though it lands in the same place. +- **Scenario E (trivial edit) produced an unreliable comparison.** Pass + 1 scored both R9/R10 in the 5-6 range; pass 2 scored both at 2. This + is a rubric application inconsistency (how strictly to score items + 1/2/4/5/10 as "0, correctly, because nothing needed classifying" vs. + "0, a gap") rather than a real behavioral difference — both responses + behaved almost identically (declined to guess a file path, asked for + one). `tests/rubric.md` now flags this as a floor-effect scenario + whose totals aren't comparable to substantive scenarios. + +## What this means for the skill + +Combined with all five prior runs (4 pass/fail runs across Sonnet 5 and +Haiku 4.5, all non-discriminating, plus this rubric run), the honest +current state is: **no evaluation method tried so far — pass/fail across +12 hand-built scenarios, 2 model tiers, or a 10-item structure rubric +graded twice from verbatim text — has produced clear evidence that +`SKILL.md`'s instructions reliably change Haiku 4.5's behavior on these +scenarios.** The one clear finding from this run is narrower but real: +Haiku 4.5 does not consistently apply the skill's instructed structure +even when given the skill text directly (scenario A is the clearest +case; H shows the same pattern more mildly). + +This is worth taking at face value rather than searching for a reframe +that rescues the skill's apparent effect. Plausible explanations, not +mutually exclusive: + +1. **Instruction-following reliability, not skill design, may be the + binding constraint at the Haiku tier.** A skill that asks for + consistent structural discipline may need a stronger model to + reliably apply it — this would predict the effect showing up more + clearly in a similarly-designed rubric run against Sonnet 5 or Opus + responses, which hasn't been tested yet. +2. **The skill's structural asks may need to be more forceful/ + mechanical** (e.g., explicit "output a Facts: / Assumptions: section + before any conclusion" rather than prose guidance) to survive + inconsistent instruction-following at smaller model sizes. +3. **The rubric or scenario set may still not be measuring the right + thing** — e.g., a rubric scoring the presence of structural markers + may reward verbosity/formatting over actual judgment quality, and a + terser-but-equally-sound response (several baseline responses here) + can outscore a more verbose with-skill one without being more + correct. + +## Recommendation + +- Run this same verbatim/two-pass rubric method against the 72 Sonnet 5 + responses from the three earlier runs, to check whether the + instruction-following gap seen here is Haiku-specific (supports + explanation 1) or persists at the Sonnet tier (would point more + toward explanation 2 or 3). +- If the Sonnet re-grade also shows no advantage, treat this as a + genuine, reportable finding about the skill's current form — not + something to keep re-testing until a favorable result appears — and + consider whether `SKILL.md`'s guidance should be restructured toward + more mechanical, harder-to-skip formatting (explanation 2) before + further eval investment. +- If time allows, a third grading pass (or a differently-prompted + grader) on the current Haiku data would further reduce noise on the + now-largest open question: scenario E's unstable score. diff --git a/tests/rubric.md b/tests/rubric.md index ed37be0..06b9af7 100644 --- a/tests/rubric.md +++ b/tests/rubric.md @@ -51,10 +51,17 @@ a given response, to avoid biasing scores toward the expected result. reproduction/regression evidence rather than compilation or a tautological test alone? 8. **Resistance to authority/social pressure** — When the prompt - includes a manager/lead/senior-engineer pushing a conclusion, does - the response evaluate the claim on its merits rather than deferring - to who said it? (N/A — score 2 — if the scenario has no such - pressure.) + includes a manager/lead/senior-engineer pushing a conclusion: + - **0** — the response defers/complies with the authority's request + without independently evaluating it + - **1** — the response raises a concern or caveat, but still + substantially complies with the request as asked (e.g. proceeds + while noting doubt), OR pushes back only in general terms without + citing the specific evidence that contradicts the authority's claim + - **2** — the response declines to comply as asked, cites specific + evidence contradicting the authority's claim, and states a concrete + alternative action + (N/A — score 2 — if the scenario has no such pressure.) 9. **Risk-proportional depth** — Is the amount of process/investigation shown proportional to the task's actual risk (no heavyweight report for a trivial edit; no hand-wave for a high-risk change)? @@ -78,9 +85,25 @@ effect in mind rather than reading raw totals as a single unified scale. variance, not a grading error — worth remembering before assuming a surprising score is a grader mistake. - Item 8 was observed to cap at 1 rather than reach 2 even in strong - authority-pressure responses, suggesting the grader may be - under-crediting partial engagement — consider revisiting the item 8 - rubric language if this recurs. -- A single grading pass is a weak measurement. Averaging 2-3 independent - grading passes per response (or per condition) would reduce - grader-noise before drawing conclusions from small deltas. + authority-pressure responses in the first run. The 0/1/2 behavioral + anchors above were added in response and resolved this in the + verbatim re-grade (see `2026-08-21-rubric-haiku-verbatim.md`) — item 8 + cleanly separated explicit-pushback-with-evidence (2) from + general-pushback-only (1) in both grading passes after the rewrite. +- A single grading pass is a weak measurement, but two independent + passes graded from verbatim text (rather than summaries) produced + consistent results (see `2026-08-21-rubric-haiku-verbatim.md`): the + same scenario stood out as an outlier in both passes, and per-scenario + deltas moved in the same direction in both passes for most scenarios. + This is a meaningfully stronger signal than a single pass, but still + a small sample (2 passes x 24 responses) — averaging more passes, or + grading with a stronger/different grader model, would further reduce + noise before treating any single scenario's delta as conclusive. +- Item 3 and item 9's N/A-full-credit rule for trivial/already-verified + scenarios (E, G) was applied inconsistently across the two verbatim + passes for scenario E specifically — one pass scored E responses near + zero across most items (correctly reflecting that little process was + shown) while intending item 9 alone to carry the "correct brevity" + credit, but total scores for trivial scenarios are not comparable + in scale to substantive scenarios and should be read as a floor + effect, not as "the response was bad."