AgentGuard Benchmark Suite

46 vulnerable + clean AI agent code samples with known OWASP ASI mappings. Use to test AgentGuard, Semgrep, CodeQL, and other SAST tools for agent-specific vulnerabilities.

What This Is

A curated collection of vulnerable AI agent code samples organized by OWASP ASI Top 10 category. Each sample is a minimal, reproducible example of a real vulnerability pattern found in production AI agent code.

Use this suite to:

• Test if your scanner detects known patterns
• Measure false positive rates against clean code
• Compare AgentGuard vs. other SAST tools
• Learn about agent-specific vulnerability patterns

Structure

samples/
  ASI01/ # Prompt Injection (6 samples)
  ASI02/ # Tool Abuse (5 samples)
  ASI03/ # Data Exfiltration (4 samples)
  ASI06/ # Insecure Output Handling (2 samples)
  ASI07/ # Credential Exposure (6 samples)
  ASI09/ # Agent Loop Exploitation (2 samples)
  ASI10/ # Trust Boundary Violation (5 samples)
  clean/ # Safe code (2 samples) - for FP testing

Running Benchmarks

With AgentGuard

pip install dfx-agentguard

# Scan all samples
for dir in samples/*/; do
  echo "=== Scanning $dir ==="
  agentguard "$dir" --format json --no-exit-code
done

# Compare results
python benchmark.py --scanner agentguard --dir samples/

Expected Results

Category	Samples	Expected Findings	AgentGuard Detection Rate
ASI01	6	6	100%
ASI02	5	5	100%
ASI03	4	4	100%
ASI06	2	2	100%
ASI07	6	6	100%
ASI09	2	2	100%
ASI10	5	5	100%
clean	2	0	0% (no false positives)

Contributing

Found a vulnerability pattern not covered? Add a sample!

• One vulnerability per file
• Name: {ASI category}_{pattern}_{language}.{ext}
• Include a comment describing the vulnerability
• Clean samples go in clean/

License

MIT

---

AgentGuard Ecosystem

AgentGuard is the core security scanner. Companion tools:

Tool	Purpose	Install
agentguard	AI agent code security scanner	pip install dfx-agentguard
mcp-scanner	MCP server security audit	pip install dfx-mcp-scanner
agentguard-app	GitHub App for PR reviews	Install from Marketplace
agentguard-vscode	VS Code inline diagnostics	Install from VS Code
agentguard-benchmark	Detection benchmark suite	git clone
agentguard-demo	Live demo with Code Scanning	git clone

19 detection rules | 102 tests | 50 benchmark samples | OWASP ASI Top 10 GitHub Action: dockfixlabs/agentguard@v1