Skip to content

Add InjecAgent and AgentDojo injection guard set #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
berkay-byte wants to merge 1 commit into
base: main
Choose a base branch
from
+1,721 −6
Open

Add InjecAgent and AgentDojo injection guard set #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,17 @@ The same showcase on ALFWorld, Blackjack and Millionaire is under `results/<eval

## The injection guard rail

The rail scores 20 of 20 on its labelled set at a median of 464 ms (`s1a run injection_guard`).
The shipped public fixture contains the ten original negative controls plus 1,054 InjecAgent cases and 629
AgentDojo security cases (1,703 records total). The source cases are retained as compact JSONL with their benchmark
name and source category in `note`; the original records remain the negative control so precision is measurable.

| labelled set | records | positive | negative | Jev precision | Jev recall |
|---|---:|---:|---:|---:|---:|
| Public fixture (`injection-public.jsonl`) | 1,703 | 1,693 | 10 | pending Jev key | pending Jev key |

Run the benchmark with `uv run s1a run injection_guard --slot jev --labelled-set
evals/labelled/injection-public.jsonl`. The source releases are [InjecAgent](https://github.com/uiuc-kang-lab/InjecAgent)
and [AgentDojo](https://github.com/ethz-spylab/agentdojo).

## Series

Expand Down
Loading