Repository navigation
fix(scanners): sort findings so the same input gives the same report - #98
Conversation
Second-model reviewOne pass by a second model over the branch at
Checked, no finding:
Verdict: request changes (finding 1). In plain EnglishSecGate could list the same problems in a different order on two runs of the same code, which made its promise of byte-identical results false. This change sorts the list, and the reviewer confirmed that the sorting happens before every report and does not change any score or verdict. The reviewer also found that the sorting rule broke down when a scanner sends an odd line number, such as a word instead of a number, so the order could still change in that rare case. That is fixed and tested in the next commit. |
Fix for finding 1:
|
No issue to close: the demo rehearsal of 2026-11-15 ran SecGate 0.3.2 twice on the same commit of juice-shop and got the same findings in a different order. No release in this PR: 0.3.3 ships later together with the documentation changes.
Cause
secgate.jsstarts the six scanner runs together (Promise.all), and each one adds its findings as soon as it has parsed its output, so the list follows the order in which the scanners finish.266448athe first difference between the two runs was at index 10, where two Trivy findings fordata/static/codefixes/iacLeakedKeyChallenge_1.tf(lines 81 and 74) had swapped.test/determinism.mjsfed its findings in one fixed order, so it could not see this.What changed
compareFindings(lib/scanners.mjs): a total order by scanner, file, line, column, rule (signature) and image, then the whole finding as text, so any two different findings always come in the same order. Locations are ordered missing first, then finite numbers, then any other value by its JSON text, and text keys by their string form, so odd scanner values cannot break the order (review finding 1,c3d83c2).secgate.jssorts the findings with it as soon as every scanner has finished, before the baseline, the JSON, HTML and SARIF reports and the console output.test/schema.mjsread the first Gitleaks finding by its position in the stub's output; it now finds it by file.docs/aggregation.md(a public page) and CHANGELOG[Unreleased]describe the order.Proof
New test in
test/determinism.mjs: two CLI runs with only stub Semgrep and Trivy on PATH. Between the runs the stubs finish in the opposite order, and Trivy lists its two results (lines 81 and 74 of one file) in the opposite order. "On main" means the branch's test file run against2e50cc2:sorted by scanner, file and line: expected ["semgrep:1","trivy:74","trivy:81"], got ["semgrep:1","trivy:81","trivy:74"]null,"1","unknown",2; files10,"9") to one listcompareFindings); atb98b966:expected 1, got 5The rehearsal's case with this branch: juice-shop at
266448a, run twice withSECGATE_TIMESTAMP=2026-01-01T00:00:00Z node secgate.js <juice-shop> --output-dir <dir>:secgate-v7-report.jsonsha25639df6c2a…ffb43a2039df6c2a…ffb43a20git statusof the targetGate
npm ci && npm run check:deps && npm run lint && npm teston Node 22.23.4 atc3d83c2, exit 0:Not changed
(tool, signature, file, image, line).