Skip to content

Remove keyword-matching component from PatternReferencesAssessor #480

@jwm4

Description

@jwm4

Problem

The PatternReferencesAssessor has three scoring components:

  1. .claude/skills/ with SKILL.md files (+60/+30 pts depending on count)
  2. Keyword regex matching in CLAUDE.md/AGENTS.md (+40 pts)
  3. Example directories (+20 pts)

Component #2 scans context files for phrases like "reference implementation", "follow the pattern in", "use X as a template" using regex. This is problematic:

  • Weak signal: It checks whether context files talk about patterns, not whether the repo actually has useful patterns
  • Fragile: The regex matches singular "reference implementation" but not plural "reference implementations", causing legitimate context files to miss points
  • Gameable: Writing "follow the pattern" without pointing to anything useful earns full credit
  • Not well motivated: The underlying insight (pattern references help agents do copy-modify) is sound, but the keyword scan doesn't verify that patterns actually exist or are useful

Proposal

Remove the keyword-matching component (#2) entirely. The assessor would score based on:

  • Skills directory with SKILL.md files (structured, machine-readable patterns)
  • Example directories (concrete reference code)

These are structural checks that verify something real exists, rather than checking for documentation about patterns.

The pass threshold (currently 40) and point values for the remaining components may need rebalancing.

Context

Discovered when slimming down AGENTS.md (#479): the old CLAUDE.md happened to match "reference implementation" (singular) for +40 points, but the rewritten AGENTS.md used "reference implementations" (plural) and lost the points. The score drop itself is fine, but it highlighted how arbitrary the keyword matching is.

Posted by Bill Murdock with assistance from Claude Code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions