Conversation
The LLM meta-analyzer adjudicated thousands of ATR regex hits, so the blocking decision flipped between runs on identical content. Follow skill-scanner 2.2.0's recommended gating setup instead: - Drop --enable-meta. Upstream measured it costing 16.4 points of recall and now disables it by default. - Drop the ATR rule pack. It produced ~98% of HIGH+ hits across the catalog, and upstream ATR marks all of its skill-targeted rules as maturity "test", not for enforcement. - Add --policy quiet, which demotes noisy rules and caps low-confidence and contextual-risk LLM findings at LOW. Update the allowlist examples in the docs, which used an ATR rule. Refs #1047 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dan Barr <6922515+danbarr@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the scan-config half of #1047.
The blocking skill scan flipped between pass and fail on identical content because the LLM meta-analyzer was deciding which of thousands of ATR regex hits to keep. This switches to skill-scanner 2.2.0's recommended gating setup:
--enable-meta. Upstream measured it costing 16.4 points of recall, and it's now off by default.maturity: test.--policy quiet. It lowers noisy rules and caps low-confidence and contextual-risk judge findings at LOW.The LLM judge, consensus runs, and the HIGH block threshold are unchanged. The docs' allowlist examples used an ATR rule and now use a PromptGuard one.
Merge before the companion allowlist PR, which adds entries for the 33 findings that block under the new settings. The smoke job doesn't scan skills, so this PR's CI doesn't exercise the new flags. I verified them locally against scanner 2.2.0, with and without the judge.
🤖 Generated with Claude Code