Skip to content

refactor(hack): port agent-skill validator to Go - #1509

Merged
skevetter merged 2 commits into
mainfrom
refactor/ci-validate-agent-skill-go
Oct 17, 2026
Merged

skevetter merged 2 commits into
mainfrom
refactor/ci-validate-agent-skill-go

Conversation

@skevetter

@skevetter skevetter commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Replace hack/ci/validate-agent-skill.sh with the Go command hack/validate_agent_skill.
  • Same steps, same order, same pins and messages. Stops at the first failing step.
  • Installed copies are compared byte for byte against the source tree, and SKILL.md must be a non-empty regular file.
  • Wire the agent-skill job in pr-ci.yml (adds setup-go) and the agent-skill:validate Taskfile task to the Go command. Update the doc paragraph that named the script.

Why

The bash script is Linux and macOS only, and it cannot be unit tested. The Go port runs on all platforms and has table-driven tests for each step, including failure at each step, the installed-tree comparison, and the venv layout per GOOS.

Behaviour changes: none on success. Two deliberate deltas: the installed SKILL.md check is stricter (non-empty regular file, not just test -f), and the npx --json output is discarded (the old script wrote it to a deleted temp file).

Known gap: the tree comparison covers regular files only. Empty directories are not compared (diff -r did).

Validation

  • gofmt -l hack/validate_agent_skill: clean
  • go vet ./hack/validate_agent_skill/...: clean
  • go test -count=1 ./hack/validate_agent_skill/...: pass
  • GOOS=linux|windows|darwin golangci-lint run ./hack/validate_agent_skill/...: 0 issues
  • go run ./hack/validate_agent_skill: passed end to end (pip install, skills-ref validate, Codex and Claude Code installs)
  • PYTHON=/nonexistent go run ./hack/validate_agent_skill: exits 1 with the Python 3.11+ message
  • go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/pr-ci.yml: no findings in the agent-skill job. One pre-existing finding on origin/main at the free-disk-space step (background: key), not touched here.

CI jobs exercised

  • agent-skill (changed; runs the new command)

CodeRabbit CLI

  • coderabbit review --agent --committed against origin/main: completed, 0 findings, 6 files reviewed.

Auto-merge

No. Changes .github/workflows/pr-ci.yml; needs human review.

Summary by CodeRabbit

  • Chores
    • Updated automated validation for the Devsy skill to check its required files, compatibility, and consistency across supported installations.
  • Documentation
    • Updated the validation guidance to reflect the current command and steps.

@skevetter skevetter added the ci-refactor CI/e2e refactor label Oct 16, 2026
@netlify

netlify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for devsydev canceled.

Name Link
🔨 Latest commit 2707cd1
🔍 Latest deploy log https://app.netlify.com/projects/devsydev/deploys/6ad27ca8ac19256dc2a335ae

@netlify

netlify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for images-devsy-sh canceled.

Name Link
🔨 Latest commit 2707cd1
🔍 Latest deploy log https://app.netlify.com/projects/images-devsy-sh/deploys/6ad27ca893f5ef7050bd3596

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f8f86b3c-7eb2-425a-b4a3-7737f870faa9
📥 Commits

Reviewing files that changed from the base of the PR and between 2a9f58e and 2707cd1.

📒 Files selected for processing: 6
  • .github/workflows/pr-ci.yml
  • Taskfile.yml
  • hack/ci/agent-skill-evaluation.md
  • hack/ci/validate-agent-skill.sh
  • hack/validate_agent_skill/main.go
  • hack/validate_agent_skill/main_test.go
💤 Files with no reviewable changes: 1
  • hack/ci/validate-agent-skill.sh

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

📝 Walkthrough

The skill validation workflow moves from a shell script to a Go CLI. The CLI checks required files, validates the skill, installs it for Codex and Claude Code, and compares each installed copy with the source. CI, Taskfile, documentation, and tests are updated.

Changes: Agent skill validation

Layer / File(s) Summary
Validator command and inputs
hack/validate_agent_skill/main.go, hack/validate_agent_skill/main_test.go
The Go CLI accepts a repository root, selects a Python executable, checks required source files, and uses OS-specific virtual-environment paths. Tests cover these checks and the executable paths.
Validation, installation, and migration
hack/validate_agent_skill/main.go, hack/validate_agent_skill/main_test.go, hack/ci/validate-agent-skill.sh, .github/workflows/pr-ci.yml, Taskfile.yml, hack/ci/agent-skill-evaluation.md
The CLI runs the pinned validator, installs the skill for both targets, and compares installed files with the source. Tests cover failures, comparisons, and success output. CI and Taskfile invoke the Go command, the documentation names it, and the shell script is removed.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Merge Risk: ⚪ Minimal · up to 2707c

The validator migration is mergeable after normal checks; no actionable validation failure remains identified.

Security architecture risk: 🔵 Low · up to 2707c

The change remains confined to repository validation tooling. Existing tool pins, CI permissions, and validation ordering are preserved, and current callers do not supply externally controlled repository arguments. No introduced security defect was established; remaining uncertainty concerns external-tool behavior and interruption cleanup.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The effective execution scope remains the calling CI runner or developer account. Downloaded validation and installation tools inherit that account’s environment and process authority; temporary installation directories do not restrict their access. This exposure predates the migration.

Security Findings and Attack Paths

  • inferred — The added repository-root option broadens explicit caller choice but does not itself grant authority. Exploiting that choice requires control of invocation arguments; the inspected CI and Taskfile callers do not forward PR metadata into it. No new attacker-controlled argument path was established.

Trust Boundaries and Controls

  • observed — The Go runner passes executable arguments directly through exec.Command rather than constructing shell commands. Existing external-tool pins, per-agent working directories, and inherited environment behavior are retained, and the CI job gains no additional declared token permissions.

Resilience and Maintainability Implications

  • observed — Failure-injection tests assert that each failed external step stops subsequent execution and produces no success output. These are mocked orchestration checks, not evidence of actual installer containment or signal-handling behavior.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 32.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 2 files. (3 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the primary change: replacing the agent-skill validator with a Go implementation.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 32.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 2 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

🔧 zizmor (1.30.1)
.github/workflows/pr-ci.yml

INFO zizmor: 🌈 zizmor v1.30.1
WARN audit:audit{input=Workflow(file://.github/workflows/pr-ci.yml)}: zizmor::audit::unsound_ternary: couldn't parse expression: matrix.provider-mode || ''parity''
fatal: no audit was performed
'template-injection' audit failed on file://.github/workflows/pr-ci.yml

Caused by:
0: error in 'template-injection' audit
1: failed to locate subfeature 'Subfeature { after: 143, fragment: Regex(Regex("matrix\.provider\-mode\s+\|\|\s+'parity'")) }' in feature ''"${RUNNER_TEMP}/devsy-ci-microsandbox" run --label "${{ matrix.label }}" --expected-count "${{ matrix.expected-spec-count }}" --provider-mode "${{ matrix.provider-mode || ''parity'' }}"''

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…agent-skill-go

# Conflicts:
#	hack/ci/agent-skill-evaluation.md
@skevetter
skevetter marked this pull request as ready for review October 16, 2026 20:37
@mergify

mergify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@skevetter
skevetter merged commit 234b5f2 into main Oct 17, 2026
111 checks passed
@skevetter
skevetter deleted the refactor/ci-validate-agent-skill-go branch October 17, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-refactor CI/e2e refactor size/xl

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant