Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions .github/workflows/skillspector.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
name: SkillSpector

on:
pull_request:
push:
branches:
- main
workflow_dispatch:

permissions:
contents: read

concurrency:
group: skillspector-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
scan:
name: Scan AI primitives
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
contents: read
security-events: write
steps:
- name: Checkout toolkit
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Test scan runner
run: python3 -m unittest discover -s tests -p 'test_skillspector.py'

- name: Checkout pinned SkillSpector
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: NVIDIA/SkillSpector
ref: 69dcdfb74487d361ba4c811d088cfdea2ff3a9dc # v2.11.2
path: .skillspector-source
persist-credentials: false

- name: Build scanner
run: docker build --tag skillspector-ci .skillspector-source

- name: Exercise the real scanner container
env:
REPORT_DIR: ${{ runner.temp }}/skillspector-reports/smoke
run: env -u GITHUB_STEP_SUMMARY python3 tests/smoke-skillspector.py --output-dir "$REPORT_DIR"

- name: Scan and enforce severity gate
id: scan
env:
REPORT_DIR: ${{ runner.temp }}/skillspector-reports/repository
run: python3 scripts/scan-skills.py --output-dir "$REPORT_DIR" --sarif

- name: Publish findings to Code Scanning
if: ${{ !cancelled() && steps.scan.outputs.sarif_created == 'true' }}
uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4.38.0
with:
sarif_file: ${{ runner.temp }}/skillspector-reports/repository/findings.sarif
category: skillspector/static
wait-for-processing: true

- name: Upload reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: skillspector-reports
path: ${{ runner.temp }}/skillspector-reports
if-no-files-found: warn
retention-days: 14
80 changes: 80 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,86 @@ Run the suite on demand with `mise exec -- lefthook run pre-commit --all-files`.
To bypass hooks in an emergency, use `git commit --no-verify` (please don't make
a habit of it).

## SkillSpector security scans

The [SkillSpector workflow](.github/workflows/skillspector.yml) runs on every
pull request, push to `main`, and manual dispatch. It scans each skill directory
under `.apm/skills/` separately, including supporting files, plus the other
`.apm/` primitive directories, committed `.github/agents/` and `.github/prompts/`,
and the `scripts/git-push-approval/` and `scripts/tool-guardian/` hook implementations.
Add any new hook implementation directories to `EXTRA_ROOTS` in
[`scripts/scan-skills.py`](scripts/scan-skills.py).

**Any individual HIGH or CRITICAL finding fails the check.** LOW and MEDIUM
findings remain visible but do not block it, regardless of the aggregate risk
score. Scanner errors, missing or invalid reports, and incomplete inspection
also fail. Every selected directory is checked for descendant symlinks and
non-regular files without following links; unreadable content fails discovery.
Reports must explicitly contain an empty `scope_exclusions` list: upstream can
call an inspection complete even after excluding files, which is not sufficient
for this gate. No exclusions, findings, or baselines are automatically accepted.

Review findings in the **Actions run summary**: it shows severity, rule, linked
repository file/line, matched evidence, explanations, and analysis limitations.
Findings and counts from incomplete scans remain visible alongside an `ERROR`
status. A dash means no readable report was available, not zero findings.
Long summaries show up to 100 entries per section, with the highest-severity
findings first; full data remains in the artifact.

Repository findings are also exported to **SARIF** and uploaded using
`github/codeql-action/upload-sarif`, even when the severity gate fails. Find them
under **Security > Code scanning**, selecting the relevant branch/PR and the
SkillSpector tool. PR annotations appear only where findings overlap changed
lines. Repository-relative locations and stable rule/severity IDs let GitHub
track findings across runs; the upload action supplies source fingerprints.
SARIF records incomplete execution and diagnostic notifications rather than
presenting partial scans as clean. Severity bands are mapped to GitHub's numeric
security-severity categories, not independently calculated CVSS scores.

The scan job grants only `contents: read` and `security-events: write`; it does
not need a PR-write token or `pull_request_target`. Only the repository scan opts
in to SARIF export. Synthetic smoke findings stay in the artifact; they are
excluded from the Actions summary and never uploaded to Code Scanning.
Uploads require a generated SARIF file and are skipped on cancellation.

The `skillspector-reports` artifact retains raw JSON reports, scanner logs,
`summary.json`, `summary.md`, and `repository/findings.sarif` for 14 days.
Review the evidence before deciding how to remediate a finding; heuristic
matches are not proof of exploitability. Reports may contain source excerpts,
so treat them with the same sensitivity as the scanned content.

The workflow builds [NVIDIA/SkillSpector](https://github.com/NVIDIA/SkillSpector)
v2.11.2 from commit `69dcdfb74487d361ba4c811d088cfdea2ff3a9dc` using its upstream
Dockerfile. Building downloads Python dependencies; the scanner revision and
upstream base image are pinned, but the upstream install resolves transitive
dependencies rather than using a frozen lockfile. Scanning then runs as a
non-root user with read-only input, no network, no host credentials, bounded
resources, and a three-minute timeout per target. `--no-llm` disables semantic
analysis; live OSV vulnerability queries and transitive downloads are unavailable
in the network-isolated container. OSV uses the scanner's offline fallback, so a
passing check is not a guarantee that a primitive or its dependencies are safe.

Before scanning the toolkit, the workflow exercises the same image and runner
with benign content, a synthetic HIGH finding, an invalid archive, and a short
timeout. These cases verify the real gate outcomes and saved diagnostics rather
than mocking Docker. Their reports live under `smoke/` in the artifact; the
toolkit's reports live under `repository/`. Smoke failures prevent the toolkit
scan from running, and diagnostics are uploaded even when a check fails.

For a local run on Linux with Docker, build the same pinned upstream image as
the workflow, then run:

```bash
python3 -m unittest discover -s tests -p 'test_skillspector.py'
python3 tests/smoke-skillspector.py --output-dir /tmp/skillspector-smoke
python3 scripts/scan-skills.py --output-dir /tmp/skillspector-reports --sarif
```

Use a new output directory for each run; existing reports are never reused.
No Python packages, scanner service, or reusable scanning skill are added to the
APM package. To enforce this check at merge time, require **Scan AI primitives**
in the repository's branch protection or ruleset.

## 🚀 Submission Process

1. **Fork** this repository
Expand Down
Loading
Loading