SME review: the three DRAFT entry mappings in evals/EXTERNAL_BENCHMARKS.md

[emmanuelgjr]

Low priority. Nothing depends on this; it exists so three DRAFT markers do not quietly harden into asserted mappings.

What needs reviewing

evals/EXTERNAL_BENCHMARKS.md catalogues three published benchmarks (added in #118). Each is listed against an OWASP entry, and that column is a reading of the benchmark's stated scope, not a reviewed mapping:

Benchmark	Entry as drafted	What it actually measures
LongPIBench	LLM01 Prompt Injection	Prompt injection in long-context settings — peer review, resume screening, code review, email summary; synthetic and real-world datasets
GenIaC-SecBench	LLM10 Improper Output Handling	Security of LLM-generated Infrastructure-as-Code against a human baseline; 100 scenarios, 12 model configurations, 1,196 artifacts
TIER	LLM01 Prompt Injection	Behavioural safety across threat implicitness; 4 risk domains × 4 threat levels, 6-label scale, 2 LLM judges

The two I would look at hardest

• GenIaC-SecBench → LLM10. Insecure generated IaC is arguably closer to LLM05 (Data and Model Poisoning) or an agentic entry depending on how the code reaches production. LLM10 was chosen because the benchmark scores the output the model hands downstream; a reviewer may disagree.
• TIER → LLM01. TIER measures refusal behaviour under jailbreak-style pressure. LLM01 is the closest existing entry, but "safety behaviour" is not the same thing as prompt injection, and this may be a case where no current entry fits and the honest answer is none.

LongPIBench → LLM01 is the least contentious of the three.

Please do not

• Remove the DRAFT markers until this is signed off — they are what stops the column being read as an assertion.
• Treat these as mappings in the crosswalk data. They live in a catalogue file, not in data/entries/, and nothing downstream consumes them.

Suggested timing

Review this in the same session as the ISO 27001 schema-v2 template, since both are judgment calls about how a source is represented rather than implementation work.

[ossumpossum]

Taking the three as asked.

LongPIBench to LLM01. Agree, no change. Long context is a condition the injection
happens under, not a separate class.

GenIaC-SecBench to LLM10. Agree, and I think the 2026 text settles it. I started out
disagreeing with this one, on the reading that Improper Output Handling is about a
consuming system failing to validate output before acting on it, which would put a
generation-rate benchmark outside it. That is the 2025 scoping and it does not apply
here. MIGRATION.md is explicit that LLM10 was renamed and "widened to cover the
insecure code that coding assistants generate at scale", and lists what it absorbed as
"insecure generated code reaching compilers, repositories, and production". GenIaC is a
direct instance of that. The DRAFT mapping is right.

Worth a scope note on the row rather than a change: GenIaC measures generation rate
against a human baseline and does not exercise a validation step, so it evidences the
entry without testing the mitigation. That distinction is useful to a reader deciding
whether a passing score means anything operationally.

On LLM05, which you also raised: agree it does not fit. The one reading that would make
it fit is generated code re-entering a training corpus, and GenIaC does not measure that.

TIER to LLM01. I would keep LLM01 with a scope note rather than move it to none, but
the note has to be narrow. My first instinct was that TIER measures refusal behaviour
rather than injection, so no entry covers it. That is wrong for one of its four threat
levels and right for the rest.

L4 is built from in-the-wild Wild Jailbreak prompts, Caesar-cipher transformations and
multilingual Aya Red-Teaming prompts. This repo already maps that squarely under LLM01:
llm-top10/LLM_MITREATLAS.md carries AML.T0054 LLM Jailbreak there, and LLM01's incident
list has six jailbreak entries, from Bing "Sydney" through many-shot jailbreaking. So L4
belongs to LLM01 by the project's own taxonomy.

L1 and L2 are explicit and euphemistic harmful requests with no adversarial instruction
at all. That is plain harmful-content refusal and I do not think any 2026 entry covers
it. L3 is the genuinely arguable one: Gemma-generated contextual prompts with manual
review, so implicit rather than adversarial. Whether disguising a harmful request in
benign context counts as injection is a judgment call I would rather flag than make for
you.

So: keep LLM01, with a note that it covers L4 cleanly, that L1 and L2 are uncovered by
any 2026 entry, and that L3 is undecided. A reader quoting a headline TIER figure is
quoting something three quarters of which is not measuring LLM01. That is a finding about
coverage rather than a reason to drop the mapping.

---

One thing I would add that is not a mapping question

TIER reports no inter-rater statistic for its two judges. No kappa, no alpha, no percent
agreement. It makes a deliberate choice instead, in section 4.3: it treats disagreement
as evaluator uncertainty and rests on conclusions that hold across both judges.

That is a defensible choice and I am not calling it a flaw. But it has a consequence the
catalogue should carry, and the paper's own numbers show it. Between the Qwen2.5 judge
and the Llama-3.1 judge, Mistral7B's attack success rate is 0.509 against 0.355, and
Llama3's is 0.190 against 0.075. The rankings hold in both. The absolute numbers move by
up to a factor of two and a half. A reader who quotes a TIER ASR figure without naming
the judge is quoting a number that does not transfer.

Concretely, for a catalogue whose stated job is to stop DRAFT markers hardening into
assertions, one short caveat line per entry would carry it:

• How labels are produced. TIER is LLM judged. GenIaC is static rule, three scanners,
  so that row is already covered by the existing column.
• Whether the labels were validated, and how. For TIER the honest line is "no
  agreement statistic; authors argue rankings are stable across judges", which tells a
  reader exactly how far the numbers travel.
• Whether results are reported per cell. TIER is 4 risk domains by 4 threat levels.
  Per-cell n is derivable, 296 per risk area evenly over four levels gives 74, but the
  published tables marginalise over risk domain, so the 16-cell cross-tab is not
  available. Worth noting where a reader would otherwise assume it.

None of that blocks the mappings. It is the same kind of judgment call you are already
making by refusing to let the DRAFT markers lapse, applied to the measurement rather than
the taxonomy.

Happy to open a PR with the scope notes and a caveat column if you want it in that form,
or leave this as review if you would rather keep the edits yours.