Skip to content

feat: harden SKILL.md evidence structure into an explicit output format #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doulos76 merged 1 commit into from
Aug 21, 2026
Merged

feat: harden SKILL.md evidence structure into an explicit output format #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,22 @@ Use deeper investigation when the task involves uncertainty, unfamiliar or legac

## Establish What Is Known

Before choosing a cause, distinguish relevant information as:
Before choosing a cause on any non-trivial diagnosis — this includes a bare stack trace, crash report, or error log presented with no other context — output this section before proposing a fix, using these exact labels:

```
FACTS:
- ...
ASSUMPTIONS:
- ...
INFERENCES:
- ...
UNKNOWNS:
- ...
CONFLICTING EVIDENCE:
- ...
CLAIMS:
- ...
```

- **FACT** — directly observed or verified
- **ASSUMPTION** — plausible but unverified
Expand All @@ -35,6 +50,8 @@ Before choosing a cause, distinguish relevant information as:
- **CONFLICTING EVIDENCE** — evidence that does not fit the current explanation
- **CLAIM** — statement from a comment, ticket, commit, document, or person that still requires verification when material

Omit a category only if it is genuinely empty for this task — do not omit the section itself. A single alarming symptom (a crash, a stack trace) does not make the cause trivial; the diagnosis is still non-trivial even when the fix that follows is a one-line change.

Do not silently turn claims or inferences into facts.

## Investigate Before Fixing
Expand Down Expand Up @@ -70,13 +87,17 @@ This is not an assumption that the code is correct. It is a search for hidden co

## Classify the Conclusion

Use one of these labels when useful:
For any non-trivial diagnosis, state the confidence label on its own line, in this exact form:

```
Conclusion: VERIFIED | STRONGLY SUPPORTED | INFERRED
```

- **VERIFIED** — directly demonstrated
- **STRONGLY SUPPORTED** — supported by multiple pieces of evidence and meaningful alternatives were weakened
- **INFERRED** — best current explanation but not sufficiently proven

Do not invent precise confidence percentages without a real measurement basis.
A fix that compiles or a plausible-looking cause is not, by itself, grounds for VERIFIED. Do not invent precise confidence percentages without a real measurement basis.

## Change Minimally

Expand Down Expand Up @@ -118,7 +139,7 @@ Check whether:

## Report Material Uncertainty

For non-trivial investigations, communicate concisely:
For non-trivial investigations, communicate concisely using this structure:

- Observed Facts
- Assumptions / Unknowns
Expand All @@ -130,11 +151,12 @@ For non-trivial investigations, communicate concisely:
- Regression Risk
- Remaining Uncertainty

Do not force this full template onto trivial work.
A stack trace, crash report, or bug description handed to you with no other context is a non-trivial investigation by default — use this structure even if the eventual fix turns out to be small. Trivial work is work that is already fully specified and directly verifiable (e.g., "change this label text," "the failing assertion already names the exact line to flip") — not work that merely *looks* urgent or has an obvious-seeming culprit. When genuinely trivial, skip the template rather than force it — but the diagnosis step (Establish What Is Known, Classify the Conclusion) still applies whenever a cause is being chosen, not just when the final report is written.

## Never

- Invent a root cause from a stack trace alone.
- Skip the FACTS/ASSUMPTIONS/UNKNOWNS breakdown for a diagnosis just because the top frame or first error message suggests an obvious cause.
- Present a plausible explanation as a verified fact.
- Search only for confirmation of the first hypothesis.
- Ignore conflicting evidence.
Expand Down
95 changes: 95 additions & 0 deletions tests/results/2026-08-21-skillmd-format-hardening.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# SKILL.md format hardening — verification — 2026-08-21

Follow-up to
[`2026-08-21-rubric-sonnet-verbatim.md`](./2026-08-21-rubric-sonnet-verbatim.md),
which found (across both Haiku 4.5 and Sonnet 5) that with-skill responses
to scenario A (Stack Trace Anchoring) and, to a lesser extent, scenario B
(Strange Legacy Code) frequently skipped the skill's own instructed
structure — jumping straight to a diagnosis/fix with no FACT/ASSUMPTION
labeling, no falsification step, no explicit confidence classification —
while unprimed baseline responses to the identical prompt often produced
that structure spontaneously. Both prior runs recommended restating the
skill's structural asks as an explicit, harder-to-skip output-format
requirement rather than prose guidance.

## Change made

`SKILL.md` was edited in three places, minimally, to convert prose
guidance into explicit format requirements:

1. **"Establish What Is Known"** now specifies an exact section format
(`FACTS: / ASSUMPTIONS: / INFERENCES: / UNKNOWNS: / CONFLICTING
EVIDENCE: / CLAIMS:`) to output before proposing a fix on any
non-trivial diagnosis, and explicitly states that a bare stack trace or
crash report is non-trivial by default even when the eventual fix is
small.
2. **"Classify the Conclusion"** now requires the confidence label on its
own line in an exact form (`Conclusion: VERIFIED | STRONGLY SUPPORTED |
INFERRED`), and adds "a fix that compiles... is not, by itself, grounds
for VERIFIED."
3. **"Report Material Uncertainty"** and **"Never"** were both updated
with an explicit anchor: a stack trace/crash/bug report with no other
context is non-trivial by default, and skipping the FACTS/ASSUMPTIONS/
UNKNOWNS breakdown because the top frame looks like an obvious cause is
now a named "Never."

No other section of `SKILL.md` was changed. The core principles, pillars,
risk-proportional depth guidance, and all other sections are untouched.

## Verification method

Re-ran scenario A and B with-skill only (baseline is unaffected by a
skill-text change, so it was not re-run) using the new `SKILL.md` text,
Sonnet 5, verbatim capture. Graded with the same rubric
(`tests/rubric.md`), single pass (not two — this is a quick verification
of a specific fix, not a full re-characterization; a full two-pass re-run
of all 12 scenarios would be needed before treating this as a complete
re-baseline).

## Result

| Scenario | Old with-skill | New with-skill (v2) | Baseline (for reference) |
|---|---|---|---|
| A | 8.0 / 20 | **19 / 20** | 16.5 / 20 |
| B | 11.0 / 20 | **20 / 20** | 16.0 / 20 |

Both scenarios went from well below baseline to above baseline. The new
with-skill responses use the required FACTS/ASSUMPTIONS/INFERENCES/
UNKNOWNS/CONFLICTING EVIDENCE/CLAIMS structure and an explicit Conclusion
line in both cases, and — notably — the structure surfaced substantive
content, not just formatting: scenario B's with-skill v2 response
correctly separated the two conflated changes in the prompt (safe dead-code
removal vs. a real security-relevant TLS-version-ceiling change) and
declined to perform the risky one without confirmation, which neither the
old with-skill response nor this scenario's baseline did as explicitly.

## Caveats

- **Single pass, two scenarios, one model tier.** This confirms the fix
addresses the specific failure mode identified (format is now followed),
but is not a full re-characterization of the skill's aggregate effect.
The other 10 scenarios were not re-run against v2 and could regress in
ways this check wouldn't catch (e.g., the new format requirement adding
unwanted ceremony to genuinely trivial tasks — scenario E is the
scenario most likely to show this if it happens).
- **n=1 per condition.** As with all prior runs in this repo, a single
response per condition per scenario is a small sample; the magnitude of
improvement here (8→19, 11→20) is large enough to be meaningful even
given that, but should still be read as a strong directional signal
rather than a precise number.
- **Not yet tested at the Haiku 4.5 tier.** The original scenario-A finding
was strongest at Haiku. This fix was designed and verified at Sonnet 5;
whether the more forceful format requirement is enough to close the gap
at a smaller/weaker model tier is untested.

## Recommendation

- Before merging, spot-check scenario E (trivial edit) against v2 to
confirm the new "a bare bug report is non-trivial by default" framing
doesn't cause over-processing on genuinely trivial tasks — this is the
most likely place for the tightened format requirement to overshoot.
- After merging, a full two-pass verbatim re-run of all 12 scenarios
against v2 (both Sonnet 5 and Haiku 4.5) would give a complete updated
picture comparable to the two prior full rubric runs, and would confirm
whether the fix generalizes tier-independently the way the scenario-A/B
regression did.
Loading