Problem
The PatternReferencesAssessor has three scoring components:
.claude/skills/ with SKILL.md files (+60/+30 pts depending on count)
- Keyword regex matching in CLAUDE.md/AGENTS.md (+40 pts)
- Example directories (+20 pts)
Component #2 scans context files for phrases like "reference implementation", "follow the pattern in", "use X as a template" using regex. This is problematic:
- Weak signal: It checks whether context files talk about patterns, not whether the repo actually has useful patterns
- Fragile: The regex matches singular "reference implementation" but not plural "reference implementations", causing legitimate context files to miss points
- Gameable: Writing "follow the pattern" without pointing to anything useful earns full credit
- Not well motivated: The underlying insight (pattern references help agents do copy-modify) is sound, but the keyword scan doesn't verify that patterns actually exist or are useful
Proposal
Remove the keyword-matching component (#2) entirely. The assessor would score based on:
- Skills directory with SKILL.md files (structured, machine-readable patterns)
- Example directories (concrete reference code)
These are structural checks that verify something real exists, rather than checking for documentation about patterns.
The pass threshold (currently 40) and point values for the remaining components may need rebalancing.
Context
Discovered when slimming down AGENTS.md (#479): the old CLAUDE.md happened to match "reference implementation" (singular) for +40 points, but the rewritten AGENTS.md used "reference implementations" (plural) and lost the points. The score drop itself is fine, but it highlighted how arbitrary the keyword matching is.
Posted by Bill Murdock with assistance from Claude Code.
Problem
The PatternReferencesAssessor has three scoring components:
.claude/skills/with SKILL.md files (+60/+30 pts depending on count)Component #2 scans context files for phrases like "reference implementation", "follow the pattern in", "use X as a template" using regex. This is problematic:
Proposal
Remove the keyword-matching component (#2) entirely. The assessor would score based on:
These are structural checks that verify something real exists, rather than checking for documentation about patterns.
The pass threshold (currently 40) and point values for the remaining components may need rebalancing.
Context
Discovered when slimming down AGENTS.md (#479): the old CLAUDE.md happened to match "reference implementation" (singular) for +40 points, but the rewritten AGENTS.md used "reference implementations" (plural) and lost the points. The score drop itself is fine, but it highlighted how arbitrary the keyword matching is.
Posted by Bill Murdock with assistance from Claude Code.