fix(file): prefilter must enumerate overlapping literal hits#3
Merged
M09Ic merged 1 commit intoJul 7, 2026
Merged
Conversation
relevantSourcesBytes selected candidate templates per line with a non-overlapping leftmost-first Find loop (start = hit.Start + 1). When one template's literal (e.g. "begin rsa private key") shares its start with — or is otherwise shadowed by — a shorter literal contributed by another loaded template (e.g. "begin"), the automaton never reported the longer literal, so that template was silently dropped from the scan. On a real multi-secret file this made only ~5 of 156 loaded rules fire (private-key, jwt, stripe, database ... all missed) while each fired correctly when loaded alone. Enumerate every literal occurrence with FindAllOverlapping, which reports all matches regardless of the automaton's leftmost match kind. Add TestScanner_Prefilter_OverlappingLiterals: shorter shadowing literals are listed before the private-key template so they win leftmost-first tie-breaking — the exact condition that used to hide the finding. Fails on the old loop, passes now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wuchulonly
pushed a commit
to chainreactors/aiscan
that referenced
this pull request
Jul 7, 2026
bump chainreactors/proton 至 471f99e(含预过滤修复):上游 AC 行级
预过滤用非重叠 leftmost-first Find 迭代挑候选模板,长字面量
("begin rsa private key")被其它已加载模板的更短字面量("begin")
遮蔽 → 多密钥文件里 156 条规则只命中约 5 条(private-key/jwt/
stripe/database 全漏)。上游已改用 FindAllOverlapping(chainreactors/proton#3)。
proton -j 输出改用 nuclei-style 连字符键(template-id/template-name),
不再直接 marshal 内部 parsers.Result 泄漏 template_id(该共享类型
被 scan 等其它数据路径依赖,不能改库 tag);新增 jsonFinding 结构体。
修复 pkg/tools/proton 4 个 E2E:DetectsAllCategories / ProtonJSON /
Pipe_CatToProtonWithExpression / Pipe_ShellToPseudoToShell。
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
relevantSourcesBytes(the per-line prefilter that selects which templates torun) enumerated literal hits with a non-overlapping leftmost-first
Findloop (
start = hit.Start + 1).When one template's literal (e.g.
begin rsa private key) shares its startwith — or is otherwise shadowed by — a shorter literal contributed by
another loaded template (e.g.
begin), the automaton never reports the longerliteral, so that template is silently dropped from the scan.
Impact is not just cosmetic: on a real multi-secret file, only ~5 of 156 loaded
rules fired (
private-key,jwt,stripe,database… all missed), eventhough each of those templates matches correctly when loaded on its own. The
more templates you load, the more get shadowed.
Fix
Enumerate every literal occurrence with
FindAllOverlapping, which reportsall matches regardless of the automaton's leftmost match kind. The change is
confined to the single prefilter function that all three scan paths
(block / line / content) already funnel through.
Test
TestScanner_Prefilter_OverlappingLiteralslists the shorter shadowingliterals before the private-key template so they win leftmost-first
tie-breaking — the exact condition that used to hide the finding. It fails on
the old loop and passes with the fix. Existing
proton/filetests stay green.🤖 Generated with Claude Code