Skip to content

ci(security): add fail-closed SkillSpector scans - #125

Draft
DevSecNinja wants to merge 3 commits into
mainfrom
devsecninja-skillspector-integration
Draft

DevSecNinja wants to merge 3 commits into
mainfrom
devsecninja-skillspector-integration

Conversation

@DevSecNinja

@DevSecNinja DevSecNinja commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add a least-privilege GitHub Actions workflow that scans authored APM primitives, committed Copilot agents/prompts, and hook implementations using NVIDIA/SkillSpector v2.11.2 pinned to 69dcdfb74487d361ba4c811d088cfdea2ff3a9dc.
  • Run static analysis in non-root, read-only, network-isolated Docker containers with bounded resources and timeouts. No LLM calls, host credentials, transitive downloads, or automatic baseline suppression.
  • Fail on individual HIGH/CRITICAL findings rather than upstream's aggregate-risk exit code; fail closed on scanner errors, malformed/missing reports, incomplete inspection, and scope exclusions.
  • Close the independent-review coverage gap: inspect every descendant without following symlinks, reject non-regular/unreadable content, and require an explicitly empty scope_exclusions list even when upstream says inspection is complete.
  • Add real-container smoke cases for benign content, a HIGH finding, invalid archive errors, and timeout handling before the repository scan. Upload smoke and repository diagnostics even on failure.
  • Publish detailed, file/line-linked Actions summaries and SARIF to GitHub Code Scanning, including findings from incomplete scans. Keep synthetic smoke findings out of the Actions summary and Code Scanning.

Validation

  • 38 regression tests pass on Linux, including real symlink, broken-link, directory-cycle, retained partial-report, location-safety, and SARIF cases. Windows passes with the POSIX-only symlink test skipped.
  • Existing primitive validation/pre-commit hooks, actionlint, yamllint, whitespace checks, and official SARIF 2.1.0 schema validation pass.
  • PR Lint workflow: passed.
  • Real SkillSpector workflow: pinned Docker image built successfully; all four real-container smoke cases passed, including non-root report writes, HIGH-severity enforcement, scanner failure diagnostics, and actual timeout exit 124. Artifact upload and download both succeeded.
  • Code Scanning upload and processing succeeded despite the failed gate. GitHub published 43 alerts: 25 HIGH, 17 MEDIUM, and 1 LOW. A second upload retained the same alert identities. No synthetic smoke alerts were published.

Findings visibility

  • Open the Actions run summary for severity, rule, source file/line links, evidence, explanations, and incomplete-analysis diagnostics. Links point to the scanned revision. Unreadable reports show unknown counts rather than misleading zeros.
  • Open Security > Code scanning, select this PR/branch and the SkillSpector tool. PR annotations appear only when findings overlap changed lines.
  • Download skillspector-reports for complete JSON, logs, Markdown, and repository/findings.sarif. The Actions summary caps each section at 100 entries; JSON and SARIF retain all findings. Synthetic smoke reports remain available only under smoke/ in the artifact.

Current toolkit scan: draft pending triage

The real repository scan inspected 19 targets: 7 PASS, 1 BLOCK, 11 ERROR. The fail-closed check is red; the infrastructure, smoke checks, and SARIF upload are not the failing steps.

  • The reports contain 43 entries: 25 HIGH, 17 MEDIUM, and 1 LOW. These are scanner findings awaiting review, not confirmed exploitable vulnerabilities.
  • .github/agents contributes 16 HIGH entries. Tool Guardian contributes nine additional HIGH entries retained from its incomplete report; the initial summary incorrectly hid these by discarding counts before checking completeness.
  • 11 targets report incomplete inspection: nine authored skills and both hook implementation directories. The downloaded reports show unresolved-reference limitations (including prose/examples interpreted as file paths) and bounded static-parser span limits. Supporting files in the Obsidian skill are present and resolved; upstream still reports other unresolved references.
  • SARIF preserves incomplete execution and the actual gate exit code. GitHub's unsuccessful tool execution, exit code 2 warning reflects those incomplete targets, not an upload failure; the findings are still published.

Keep this PR draft until existing findings and upstream inspection limitations are triaged. No findings, exclusions, or incompleteness errors have been suppressed, and the requested HIGH/CRITICAL blocking policy has not been weakened.

Operational notes

Reports are retained for 14 days and can contain source excerpts. Building the pinned upstream Dockerfile downloads its Python dependencies; transitive dependency resolution is not frozen by upstream. Live OSV queries are unavailable during scanning, so its offline fallback is used. A passing static scan is not a safety guarantee.

The job grants only contents: read and security-events: write; no PR-write token or pull_request_target is used. GitHub security-severity values represent categorical severity bands, not independently calculated CVSS scores.

Making Scan AI primitives a required merge check is a separate branch-protection/ruleset change, not performed here. No reusable scanning skill, MCP service, APM dependency, finding baseline, or Renovate tracking changes are added; those remain outside this change's scope.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

Keep synthetic smoke cases out of the Actions summary and record the actual gate exit code in SARIF.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants