Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,28 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "Edit|Write",
"hooks": [
{
"type": "command",
"command": "black --quiet \"$CLAUDE_FILE_PATH\" 2>/dev/null; isort --quiet \"$CLAUDE_FILE_PATH\" 2>/dev/null; true"
}
]
}
],
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "echo \"$CLAUDE_TOOL_INPUT\" | grep -qE 'rm -rf|DROP TABLE|--force' && echo 'BLOCK: destructive command' && exit 1 || true"
}
]
}
]
},
"permissions": {
"allow": [
"Skill(frontend-design:frontend-design)",
Expand Down
5 changes: 3 additions & 2 deletions .claude/skills/test-assess/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ allowed-tools:
- Bash(git clone *)
- Bash(gh *)
- Bash(yes *)
- Bash(rm -rf /tmp/agentready-test-*)
- Bash(find /tmp/agentready-test-* -delete)
- Bash(rmdir /tmp/agentready-test-*)
- Bash(PYTHONPATH=src python -m agentready *)
---

Expand Down Expand Up @@ -90,7 +91,7 @@ Summarize all results in a table at the end if multiple repos were tested.
After the user has seen the results, delete the temp directory:

```bash
rm -rf $TESTDIR
find $TESTDIR -delete
```

Tell the user the cleanup is done. If any repo's reports are worth preserving,
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,7 @@ docs/_site/

# Claude
.claude/settings.local.json

# Other AI agent tooling
.agents/
.codex/
13 changes: 7 additions & 6 deletions docs/attributes.md
Original file line number Diff line number Diff line change
Expand Up @@ -794,18 +794,19 @@ Automated code quality checks before commits (pre-commit hooks) and in CI/CD pip

#### Why It Matters

Pre-commit hooks give immediate local feedback. They can be bypassed with `--no-verify`, which is why CI matters too — but for agent-generated commits that go through a normal PR flow, hooks are the first line of defense. Catching a lint error before a commit beats catching it in CI review.
Agent hooks (`.claude/settings.json`) are deterministic for agent workflows: they always execute and cannot be bypassed. Git hooks (pre-commit, Husky) provide local feedback but can be bypassed with `--no-verify`. Both matter, but agent hooks score higher because they are the primary enforcement mechanism for AI-assisted development.

#### Measurable Criteria

The assessor scores on a 100-point scale:

- **`.pre-commit-config.yaml` present** (60 pts): pre-commit hooks configured
- **`.husky` directory with hook scripts** (60 pts): Husky git hooks configured (e.g., pre-commit, commit-msg)
- **`.claude/settings.json` with hooks** (60 pts): Deterministic agent hooks configured (cannot be bypassed)
- **`.pre-commit-config.yaml` present** (40 pts): Pre-commit git hooks configured (bypassable with `--no-verify`)
- **`.husky` directory with hook scripts** (40 pts): Husky git hooks configured (bypassable with `--no-verify`)
- **`.claude/settings.json` without hooks** (10 pts): Agent settings present but no hooks defined
- **`.husky` directory without hook scripts** (10 pts): Husky directory exists but no hooks defined
- **`.claude/settings.json` with hooks** (30 pts): Claude Code hook configuration present

**Pass threshold**: 60 points or higher. Either `.pre-commit-config.yaml` or `.husky` with hook scripts is sufficient to pass.
**Pass threshold**: 40 points or higher. Any single enforcement mechanism (agent hooks, pre-commit, or Husky with scripts) is sufficient to pass.

#### Remediation

Expand Down Expand Up @@ -1032,7 +1033,7 @@ setup:
**File Size Limits** (`file_size_limits`, 3%) — Files under threshold to keep context manageable
**Separation of Concerns** (`separation_of_concerns`, 3%) — Clean module boundaries and single-responsibility
**Pattern References** (`pattern_references`, 3%) — Documented patterns for common changes. Skills scoring is tiered: 1-2 SKILL.md files earn partial credit (30 pts), 3+ earn full credit (60 pts). Context files >150 lines without skills trigger a warning
**Design Intent Documentation** (`design_intent`, 3%) — Preconditions, invariants, and rationale in design docs (moved from T3)
**Design Intent Documentation** (`design_intent`, 3%) — Preconditions, invariants, and rationale in design docs (moved from T3). Enforcement bonus: advisory rules in AGENTS.md requiring design doc updates (+10 pts), or deterministic enforcement via hooks/skills (+15 pts). The higher of the two is awarded, not both

*Full details for each attribute available in the [research document](https://github.com/ambient-code/agentready/blob/main/RESEARCH_REPORT.md).*

Expand Down
109 changes: 109 additions & 0 deletions src/agentready/assessors/patterns.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,13 @@ def assess(self, repository: Repository) -> Finding:
evidence.append(f"Design intent language found in {filename}")
break

enforcement_pts, enforcement_evidence = self._check_design_enforcement(
repository
)
if enforcement_pts > 0:
score += enforcement_pts
evidence.extend(enforcement_evidence)

score = min(score, 100.0)

if score >= 50:
Expand All @@ -315,6 +322,99 @@ def assess(self, repository: Repository) -> Finding:
error_message=None,
)

def _check_design_enforcement(
self, repository: Repository
) -> tuple[float, list[str]]:
"""Check for enforcement of design doc updates alongside code changes.

Advisory enforcement (10 pts): AGENTS.md/CLAUDE.md rules requiring
design doc updates with architectural changes.
Deterministic enforcement (15 pts): Hooks or skills that check for
design doc updates. Awards the higher of the two, not both.
"""
import json

doc_ref_pattern = re.compile(
r"design\s+doc|docs/design|architecture\s+doc|\.adr|design\s+document",
re.IGNORECASE,
)
enforcement_verb_pattern = re.compile(
r"update|review|create|maintain|must|required|ensure|check",
re.IGNORECASE,
)

deterministic_score = 0.0
deterministic_evidence = []

settings_path = repository.path / ".claude" / "settings.json"
if settings_path.exists():
try:
settings = json.loads(settings_path.read_text(encoding="utf-8"))
hooks = settings.get("hooks", {})
hooks_str = json.dumps(hooks).lower()
if (
hooks
and doc_ref_pattern.search(hooks_str)
and enforcement_verb_pattern.search(hooks_str)
):
deterministic_score = 15.0
deterministic_evidence.append(
".claude/settings.json hooks reference design docs (deterministic enforcement)"
)
except (json.JSONDecodeError, OSError):
pass

if deterministic_score == 0:
skills_dir = repository.path / ".claude" / "skills"
if skills_dir.exists() and skills_dir.is_dir():
try:
for skill_dir in skills_dir.iterdir():
if not skill_dir.is_dir():
continue
skill_md = skill_dir / "SKILL.md"
if not skill_md.exists():
continue
try:
content = skill_md.read_text(encoding="utf-8")
if doc_ref_pattern.search(
content
) and enforcement_verb_pattern.search(content):
deterministic_score = 15.0
deterministic_evidence.append(
f".claude/skills/{skill_dir.name}/ references design doc enforcement (deterministic)"
)
break
except (OSError, UnicodeDecodeError):
continue
except OSError:
pass

if deterministic_score > 0:
return deterministic_score, deterministic_evidence

advisory_score = 0.0
advisory_evidence = []
context_files = ["AGENTS.md", "CLAUDE.md", ".claude/CLAUDE.md"]
for filename in context_files:
filepath = repository.path / filename
if not filepath.exists():
continue
try:
content = filepath.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue

if doc_ref_pattern.search(content) and enforcement_verb_pattern.search(
content
):
advisory_score = 10.0
advisory_evidence.append(
f"{filename} contains design doc update rules (advisory enforcement)"
)
break

return advisory_score, advisory_evidence

def _create_remediation(self) -> Remediation:
return Remediation(
summary="Document design intent: preconditions, invariants, and rationale",
Expand All @@ -323,11 +423,14 @@ def _create_remediation(self) -> Remediation:
"For each critical module, document preconditions, invariants, and rationale",
"Use an AI agent to reverse-engineer initial design docs from code, then enrich with intent",
"Reference design docs from CLAUDE.md/AGENTS.md",
"Add a rule to AGENTS.md requiring design doc updates with architectural changes",
"For stronger enforcement, add a hook or skill that checks for design doc updates",
],
tools=[],
commands=["mkdir -p docs/design"],
examples=[
"# docs/design/event-system.md\n## Invariants\n- Event log is append-only; never mutate or delete entries\n- Events are processed exactly-once via idempotency keys\n\n## Preconditions\n- Auth middleware must validate token before event handlers run\n\n## Rationale\n- Polling instead of webhooks: upstream API has 5s delivery SLA, too slow for our use case",
"# AGENTS.md - Advisory enforcement\n## Design Documentation\nWhen modifying component boundaries, data flows, or API contracts,\nreview and update the corresponding design doc in docs/design/.",
],
citations=[
Citation(
Expand All @@ -336,6 +439,12 @@ def _create_remediation(self) -> Remediation:
url="",
relevance="Agents cannot infer design intent from code alone",
),
Citation(
source="Red Hat",
title="Repository Scaffolding for AI Coding Agents, Section 2.3 Practice C",
url="",
relevance="Enforce design doc updates as part of architectural changes",
),
],
)

Expand Down
33 changes: 22 additions & 11 deletions src/agentready/assessors/testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -791,18 +791,22 @@ def assess(self, repository: Repository) -> Finding:
score = 0.0

if precommit_config.exists():
score += 60.0
evidence.append(".pre-commit-config.yaml found (pre-commit hooks)")
score += 40.0
evidence.append(".pre-commit-config.yaml found (git hooks, bypassable)")

if claude_settings.exists():
try:
import json

content = json.loads(claude_settings.read_text())
if "hooks" in content:
score += 30.0
hooks = content.get("hooks")
has_configured_hooks = isinstance(hooks, dict) and any(
isinstance(entries, list) and entries for entries in hooks.values()
)
if has_configured_hooks:
score += 60.0
evidence.append(
".claude/settings.json has hooks configured (agent hooks)"
".claude/settings.json has hooks configured (deterministic agent hooks)"
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
else:
score += 10.0
Expand Down Expand Up @@ -840,16 +844,18 @@ def assess(self, repository: Repository) -> Finding:
hook_scripts = []
evidence.append(".husky directory exists but could not be read")
if hook_scripts:
score += 60.0
score += 40.0
hooks_list = ", ".join(sorted(hook_scripts))
evidence.append(f".husky directory found with hooks: {hooks_list}")
evidence.append(
f".husky directory found with hooks: {hooks_list} (git hooks, bypassable)"
)
else:
score += 10.0
evidence.append(".husky directory found but no hook scripts")

score = min(score, 100.0)

if score >= 60:
if score >= 40:
return Finding(
attribute=self.attribute,
status="pass",
Expand Down Expand Up @@ -888,9 +894,9 @@ def _create_remediation(self) -> Remediation:
return Remediation(
summary="Set up deterministic enforcement with hooks and lint rules",
steps=[
"Configure .claude/settings.json with agent hooks (deterministic, cannot be bypassed)",
"Start with 2 hooks: auto-format on edit + block destructive operations",
"Install pre-commit (Python) or Husky (Node.js) for git hooks",
"Configure .claude/settings.json with agent hooks for team-wide sharing",
"Optionally add pre-commit (Python) or Husky (Node.js) for git hooks",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
"Add lint rules for import restrictions and architectural boundaries",
],
tools=["pre-commit", "husky"],
Expand All @@ -906,7 +912,12 @@ def _create_remediation(self) -> Remediation:
"PostToolUse": [
{
"matcher": "Edit|Write",
"command": "npx prettier --write $CLAUDE_FILE_PATH 2>/dev/null || true"
"hooks": [
{
"type": "command",
"command": "npx prettier --write $CLAUDE_FILE_PATH 2>/dev/null || true"
}
]
}
]
}
Expand Down
Loading
Loading