Hi — I run a small OSS project (Driftproof) that re-tests agent skills across model releases: the same eval suite with and without the skill, judge sampled 5x so every score carries a confidence band, and a drift verdict only when bands separate AND the mean moves at least a 0.05 effect floor. Four skills from this repo have been in the series since Report 001; the new Report 003 covers the opus-4-8 → opus-5 boundary. Their results:
- crafting-effective-readmes — the most interesting row in the report: a sign flip. On opus-4-8 it was net-negative (with_skill 0.760 vs baseline 0.870); on opus-5 it flipped to net-positive (0.871 vs 0.795) and IMPROVED with two band-separated cases. Anyone who benched it on the old model and dropped it would be wrong on the new one.
- commit-work — within noise, but the baseline caught up (0.847 → 0.886, now slightly above with_skill 0.882): opus-5 does Conventional Commits discipline largely unprompted, so on that model the skill currently reads as context cost. (On the sonnet boundary it was where we caught sonnet-5 writing feat: for a bug fix — Report 001.)
- writing-clearly-and-concisely — regressed on exactly one case, sitting exactly on our 0.05 minimum-effect floor: the weakest possible signal, flagged as such.
- naming-analyzer — regressed on one case, driven as much by variance widening as mean drop; the skill still nets +0.068 lift on opus-5 overall.
Full tables, per-case drivers, bands, and receipts: https://driftproofhq.com/reports/003/ — earlier boundaries: https://driftproofhq.com/reports/001/ (sonnet-4-6 → sonnet-5) and https://driftproofhq.com/reports/002/ (Claude vs GPT, where two of your skills were substrate-dependent).
The suites are Apache-2.0 and grounded in each SKILL.md's own claims — every rubric criterion must trace to text the skill actually contains: https://github.com/driftproofhq/driftproof/tree/main/suites. Yours to adopt upstream if useful; npx driftproof run re-runs them on any model with your own key. And if you think any case tests beyond what a skill actually claims, I would genuinely like to know — fairness of the exams matters more to me than the verdicts.
Hi — I run a small OSS project (Driftproof) that re-tests agent skills across model releases: the same eval suite with and without the skill, judge sampled 5x so every score carries a confidence band, and a drift verdict only when bands separate AND the mean moves at least a 0.05 effect floor. Four skills from this repo have been in the series since Report 001; the new Report 003 covers the opus-4-8 → opus-5 boundary. Their results:
Full tables, per-case drivers, bands, and receipts: https://driftproofhq.com/reports/003/ — earlier boundaries: https://driftproofhq.com/reports/001/ (sonnet-4-6 → sonnet-5) and https://driftproofhq.com/reports/002/ (Claude vs GPT, where two of your skills were substrate-dependent).
The suites are Apache-2.0 and grounded in each SKILL.md's own claims — every rubric criterion must trace to text the skill actually contains: https://github.com/driftproofhq/driftproof/tree/main/suites. Yours to adopt upstream if useful;
npx driftproof runre-runs them on any model with your own key. And if you think any case tests beyond what a skill actually claims, I would genuinely like to know — fairness of the exams matters more to me than the verdicts.