test: re-grade rubric from verbatim text, two independent passes - #15
Merged
Conversation
Addresses all three limitations flagged in the first rubric run (2026-08-21-rubric-haiku.md): grades from full verbatim response text instead of summaries, uses two independent blind grading passes averaged per response, and rewrites item 8 (authority resistance) with explicit 0/1/2 behavioral anchors to fix the mis-calibration that capped it at 1 in the first run. Result: reproduces the first run's finding under stronger method — no with-skill advantage (with-skill 11.50/20 vs. baseline 12.79/20), and scenario A's gap widened rather than narrowed (-8 -> -11). Both independent passes agree the with-skill scenario-A response asserts a diagnosis immediately with no evidence separation or hedging, while the baseline response on the identical prompt spontaneously produces the structure SKILL.md asks for. This points to an instruction- following reliability gap at the Haiku 4.5 tier rather than a grading artifact — item 8's rewrite is independently confirmed working (cleanly separates explicit-pushback-with-evidence from general-pushback in both passes). Recorded in tests/results/2026-08-21-rubric-haiku-verbatim.md, including three non-exclusive explanations (instruction-following reliability at this model tier, insufficiently mechanical skill guidance, or a rubric that may reward verbosity) and a recommendation to re-run this method against the 72 Sonnet 5 responses to check whether the gap is Haiku-specific before concluding anything about the skill's design.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #14 (first rubric run, graded from summarized text in a single pass, found no with-skill advantage — dominated by one outlier). This addresses all three limitations that run flagged:
tests/rubric.mdwith explicit 0/1/2 behavioral anchors, fixing a mis-calibration where it capped at 1 even for strong pushback.Key finding
Reproduces the first run's result under a stronger method. With-skill mean 11.50/20 vs. baseline 12.79/20 — still no with-skill advantage. Scenario A's gap widened under the stronger method (-8 -> -11), and both independent passes agree on why: the with-skill response asserts a diagnosis immediately with no evidence separation, no hedging, no falsification step, while the baseline response to the identical prompt (no skill instructions at all) spontaneously produces exactly the FACT/ASSUMPTION/UNKNOWN structure
SKILL.mdasks for. This isn't grader noise — it's Haiku 4.5 inconsistently applying the skill's own instructed structure even when given the skill text directly.Item 8's rewrite is independently confirmed working — it cleanly separated explicit-pushback-with-evidence (score 2) from general-pushback-only (score 1) in both passes on the authority-pressure scenarios (I, J, K).
Full breakdown, per-scenario deltas, and three non-exclusive explanations (instruction-following reliability at this model tier, guidance not mechanical enough, or a rubric that may reward verbosity) are in
tests/results/2026-08-21-rubric-haiku-verbatim.md.Recommendation
SKILL.mdtoward more mechanical formatting) rather than re-testing until a favorable result appears.Test plan
scripts/validate_skill.pypasses locally🤖 Generated with Claude Code