Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: validate

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
validate:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.13"]
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
- name: Install PyYAML
run: python -m pip install pyyaml==6.0.2

Check warning on line 25 in .github/workflows/validate.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Omitting "--only-binary :all:" can lead to the execution of setup scripts. Make sure it is safe here.

See more on https://sonarcloud.io/project/issues?id=OMG-Software_adversarial-code-audit-skill&issues=AaHNAkGo-DSAMddoZxEJ&open=AaHNAkGo-DSAMddoZxEJ&pullRequest=3
- name: Validate skill structure
run: python scripts/validate_skill.py .
- name: Test the helper script
run: python -m unittest discover -s scripts -p "test_*.py"
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Local tool state
.serena/
__pycache__/
*.pyc

# Audit output written beside or inside a checkout
security-audit/
44 changes: 44 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Changelog

## 0.11.0 – 2026-11-11

Result of a full review of the skill. The evidence bar and the execution-consent model are unchanged.

### Fixed

- **Output directory.** The default was written as `./security-audit/...` and described as outside the target, which is false when the agent starts in the target. SKILL.md section 4 now gives a resolution order, and a self-ignoring `.gitignore` for the case where the user chooses a directory inside the target.
- **Open findings disappeared from later reports.** An earlier confirmed finding was hidden from hunters, re-verified only if its source changed, and absent from a report built from the new run's files. Phase 1.9 now carries every open finding forward, phase 6 gives it a verdict against the current source, and a new status `fixed` records a finding that the code no longer has.
- **Git hardening was defined but not used.** The commands in phase 1 and phase 2 now use the safe form from execution.md section 1. That section also covers `git status`, tools that call Git themselves, embedded bare repositories and a first read of `.git/config`.
- **Domain-file sections were not tied to phases.** SKILL.md section 5 maps each section to the phase that reads it. "Recon additions" now go to a new file, `domain-notes.md`, so `threat-model.md` can keep its word limit. All 33 domain files point there.
- **Tool rules.** Toolchain launchers that download or run what a target file names (rustup, corepack, `yarnPath`, `go`, version managers) count as target-controlled. Target ignore files that narrow a scanner's results are leads. `gitleaks` runs with `--redact`. The container recipe runs as the user's own unprivileged user and limits file size.
- **Safety rules 6 and 9** overlapped and disagreed on patches. Rule 6 now holds the whole rule: a patch is a new file in the output directory. Rule 9 is new: urgent discoveries are reported at once.
- **Undefined terms.** Phase 6.3 defines the three claim types that need a reproduction: memory-safety, race and parser-differential.
- **Chains.** Phase 6 verifies component findings before chains, and a chain with a rejected or fixed link is rejected unless another primitive gives the same step.
- **Dangling references** to a CWE catalogue and an export script that did not exist are gone. The export script now exists.
- Phase 1.9 listed `needs-validation` as a unit status. It is a finding status.

### Added

- `assets/plan-template.md`: a layout for `plan.md`, with a progress log that a resumed run reads first.
- Scoped Full audits for a subsystem, a directory or one domain.
- `root-cause-at` and `first-seen` in the finding template. `root-cause-at` is the key for finding duplicates; phases 3, 4 and 5 compare it.
- A checkpoint after phase 1 (phase-recon.md section 1.12): the agent shows the size of the run and asks the owner about external assumptions. An owner-attested fact can settle a finding (phase-verify.md section 6.3).
- A definition of where dependency behaviour may come from, and permission to read the pinned dependency's source or documentation.
- A countable definition of "budget". A resumed run resets orphaned `in-progress` units.
- Windows: host guards for PowerShell, and a Windows Sandbox recipe.
- `scripts/audit_run.py` with `check`, `stats` and `export` (JSON and SARIF), and unit tests.
- The validator checks that SKILL.md lists every reference file, that each reference file has the standard sections, and that the product-type table names real domains.
- Evaluations: `secret-handling`, `memory-safety-source-only`, `scanner-triage`, `external-assumption`, `full-mini`, a list of trigger queries, and `evaluations/tools/workspace_inventory.py` for the mechanical checks.
- CI workflow, `.gitignore`, this changelog, `license` in the frontmatter.

### Not done

- The five new evaluation cases have not been run against an agent yet.
- The Windows Sandbox recipe and the container user flags are untested (see "Known limits" in the README).
- The version ranges in the language files were not refreshed, and no new language files were written.
- No CWE catalogue is bundled, so the validator does not check CWE numbers.
- The 45 domain and language files that the review did not read line by line still need a fact check.

## 0.10.0 and earlier

See the Git history.
32 changes: 26 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ A finding here is a demonstrated boundary failure: a lower-trust actor crosses a

## What it does

- **Three modes.** *Answer* explains methodology without touching files. *Focused* checks one finding, file, function, domain or diff, including PR regression reviews. *Full* runs the complete audit workflow over a codebase.
- **Three modes.** *Answer* explains methodology without touching files. *Focused* checks one finding, file, function or diff, including PR regression reviews. *Full* runs the complete audit workflow over a codebase, or over one subsystem or one domain with a restricted scope.
- **A phased workflow** that keeps all state in files, so any agent can stop and resume:
0. Setup → 1. Recon and threat model → 2. Lead generation → 3. Hunt → 4. Coverage critique → 5. Chain → 6. Verify → 7. Report.
- **Independent verification.** The agent that verifies a finding is not the one that found it. Every candidate goes through a disprove pass before it can be `confirmed`.
- **Explicit evidence levels.** E1 (source-traced), E2 (tool-supported) and E3 (reproduced). Memory-safety, race and parser-differential claims need E3. Other source-only confirmations carry a written reproduction waiver. Findings that depend on facts outside the repository stay `needs-validation`, without a severity.
- **Domain and language guidance.** 33 domain files (authz, injection, memory safety, AI/LLM agents, federation, smart contracts and more) and 14 language files (C, C++, Rust, Go, Python, JS/TS, JVM, .NET, PHP, Ruby, Swift/Objective-C, Lua, shell, SQL). Recon selects only the ones the code needs.
- **Capability tiers.** The skill works with read-only access (T0) and gets stronger with a shell (T1), target execution (T2) and sub-agents. The tier changes how evidence is gathered, never the evidence bar.
- **Two questions, then it works alone.** Before it starts, the skill asks how target code may run and which read-only tools it may install. After recon, it shows the size of the run and asks you about facts it cannot see in the repository, such as what your reverse proxy does. Your answers can settle findings that would otherwise stay `needs-validation`.
- **Runs build on each other.** A later run carries every open finding forward, verifies it against the current code, and reports it as still open or as `fixed`.

## Safety model

Expand All @@ -21,11 +23,14 @@ A finding here is a demonstrated boundary failure: a lower-trust actor crosses a
- **You choose how target code runs.** Builds, tests, build hooks, plugins and executable tool configuration all count as running target code. The skill detects what is available, explains the risks, and asks: a full sandbox (recommended), partial isolation such as WSL, the host with guards (last resort), or no execution.
- Nothing is downloaded or installed without your approval.
- Reproductions stop at the minimum effect: a wrong result, a dummy record or a sanitizer report. No weaponised exploits.
- Secret values are never copied into audit output, and the target source is never modified.
- Secret values are never copied into audit output, and the target source is never modified. A fix is described in its finding, or written as a patch file in the audit output if you ask for one.
- A credential that looks live, signs of an existing compromise, or a critical finding is reported to you at once, not at the end.

## Installation

Copy or clone this repository into your agent's skills directory, for example `~/.claude/skills/adversarial-code-audit/` for Claude Code. The skill needs only `SKILL.md`, `references/` and `assets/`.
Copy `SKILL.md`, `references/`, `assets/` and `scripts/audit_run.py` into your agent's skills directory, for example `~/.claude/skills/adversarial-code-audit/` for Claude Code. `audit_run.py` is an optional helper that needs Python 3.10 or later: it checks a finished run, counts coverage and exports JSON and SARIF.

Leave `evaluations/` out of an installed copy. It holds the scoring rubric, and fixtures that contain hostile instructions on purpose.

## Usage

Expand All @@ -36,7 +41,7 @@ Start your agent in the codebase and ask, for example:
- "Review this PR for security regressions."
- "How do I spot SQL injection during source review?"

Full audits write their state and `REPORT.md` to `./security-audit/<repo-name>/run-<N>/`, outside the target repository.
Full audits write their state and `REPORT.md` to `security-audit/<repo-name>/run-<N>/` in the directory that contains the target repository, not inside it. You can name another location. The files describe vulnerabilities, so treat them as confidential.

## Repository layout

Expand All @@ -48,16 +53,31 @@ Full audits write their state and `REPORT.md` to `./security-audit/<repo-name>/r
| `references/execution.md` | Isolation levels, sandbox detection and recipes, cleanup rules |
| `references/domain-*.md`, `references/lang-*.md` | Domain and language guidance |
| `assets/finding-template.md` | Template for one finding file per root cause |
| `assets/plan-template.md` | Template for `plan.md`, the control record of a run |
| `scripts/audit_run.py` | Optional helper: checks a run directory, counts coverage, exports JSON and SARIF |
| `evaluations/` | Behavioural test cases for maintainers (not part of an audit) |
| `scripts/validate_skill.py` | Checks frontmatter and local links |
| `scripts/validate_skill.py` | Checks frontmatter, local links and the shape of the reference files |
| `scripts/test_audit_run.py` | Unit tests for the helper |
| `CHANGELOG.md` | What changed in each version |

## Maintaining the skill

```bash
python scripts/validate_skill.py .
```

The validator needs PyYAML and checks structure only. For behaviour, run the cases in [evaluations/README.md](evaluations/README.md) with a fresh agent. Keep `evaluations/evaluator-only/` out of the evaluated agent's workspace.
```bash
python -m unittest discover -s scripts -p "test_*.py"
```

The validator needs PyYAML and checks structure only. Both commands run in CI on every pull request. For behaviour, run the cases in [evaluations/README.md](evaluations/README.md) with a fresh agent. Keep `evaluations/evaluator-only/` out of the evaluated agent's workspace.

## Known limits

- The Windows Sandbox recipe and the container user flags in `references/execution.md` follow the tools' documentation. They have not been run as part of this project. The skill tells the agent to check each sandbox control before it relies on one.
- The version ranges in the language files stop at about mid-2025. The skill treats a version-specific statement about a newer release as a lead to check, not as a fact.
- No language file exists yet for Dart, Elixir, Erlang, Perl, Solidity or Zig. The audit continues without one and says so.
- The evaluations do not yet cover resuming a run, carrying findings across runs, chains, or real target execution in a sandbox.

## Inspiration

Expand Down
Loading
Loading