Not a gotcha, your README already says smoke-test grade, 1-4 runs per cell, LLM judges. This is about the "LLM judges" half, with a number.
I measured the cost of that from the other side. My compression pipeline self-graded 96% on claim preservation, same model family grading its own output. An independent judge from a different family and provider scored the same pipeline at 78.4%. 17.6 points of inflation. And the independent judge is still an LLM, so it's corroboration, not proof.
What fixed it was a deterministic floor under the judged metric. Anchor facts (numbers, dates, hex ids, filenames) surviving exact-token match. No model opinion in the number. Reproduces by grep in under a second, no network, no key.
s7 is the right shape for this. Each planted fraud could carry a grep-checkable signature: a specific line that must appear in the diff, a specific test that must fail. Then the 5/5 has a floor under it that doesn't depend on Haiku's opinion of Haiku, and the trap suite becomes gradeable by someone who can't read the code (which your German reviewer is, and she said the fixtures were the most valuable part of the repo).
Your v1 to v3 table is the most useful thing in here, by the way. Rules in prose lists 1 of 4, the same rule as a forced artifact 4 of 4. That generalizes well past this repo and it's why I'm suggesting an artifact rather than a rule.
Prereg and the null, if useful: https://github.com/8889-coder/prereg-compression-null
Not a gotcha, your README already says smoke-test grade, 1-4 runs per cell, LLM judges. This is about the "LLM judges" half, with a number.
I measured the cost of that from the other side. My compression pipeline self-graded 96% on claim preservation, same model family grading its own output. An independent judge from a different family and provider scored the same pipeline at 78.4%. 17.6 points of inflation. And the independent judge is still an LLM, so it's corroboration, not proof.
What fixed it was a deterministic floor under the judged metric. Anchor facts (numbers, dates, hex ids, filenames) surviving exact-token match. No model opinion in the number. Reproduces by grep in under a second, no network, no key.
s7 is the right shape for this. Each planted fraud could carry a grep-checkable signature: a specific line that must appear in the diff, a specific test that must fail. Then the 5/5 has a floor under it that doesn't depend on Haiku's opinion of Haiku, and the trap suite becomes gradeable by someone who can't read the code (which your German reviewer is, and she said the fixtures were the most valuable part of the repo).
Your v1 to v3 table is the most useful thing in here, by the way. Rules in prose lists 1 of 4, the same rule as a forced artifact 4 of 4. That generalizes well past this repo and it's why I'm suggesting an artifact rather than a rule.
Prereg and the null, if useful: https://github.com/8889-coder/prereg-compression-null