Skip to content

fix(scanners): sort findings so the same input gives the same report - #98

Merged
danieloceno merged 2 commits into
mainfrom
fix/stable-finding-order
Nov 16, 2026
Merged

danieloceno merged 2 commits into
mainfrom
fix/stable-finding-order

Conversation

@danieloceno

@danieloceno danieloceno commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

No issue to close: the demo rehearsal of 2026-11-15 ran SecGate 0.3.2 twice on the same commit of juice-shop and got the same findings in a different order. No release in this PR: 0.3.3 ships later together with the documentation changes.

Cause

  • secgate.js starts the six scanner runs together (Promise.all), and each one adds its findings as soon as it has parsed its output, so the list follows the order in which the scanners finish.
  • Trivy does not list its results in a fixed order either. On juice-shop at 266448a the first difference between the two runs was at index 10, where two Trivy findings for data/static/codefixes/iacLeakedKeyChallenge_1.tf (lines 81 and 74) had swapped.
  • So the JSON report differed between two runs on the same input, while the README's accuracy contract says "same inputs produce JSON-byte-identical findings". test/determinism.mjs fed its findings in one fixed order, so it could not see this.

What changed

  • compareFindings (lib/scanners.mjs): a total order by scanner, file, line, column, rule (signature) and image, then the whole finding as text, so any two different findings always come in the same order. Locations are ordered missing first, then finite numbers, then any other value by its JSON text, and text keys by their string form, so odd scanner values cannot break the order (review finding 1, c3d83c2).
  • secgate.js sorts the findings with it as soon as every scanner has finished, before the baseline, the JSON, HTML and SARIF reports and the console output.
  • test/schema.mjs read the first Gitleaks finding by its position in the stub's output; it now finds it by file.
  • README accuracy contract, docs/aggregation.md (a public page) and CHANGELOG [Unreleased] describe the order.

Proof

New test in test/determinism.mjs: two CLI runs with only stub Semgrep and Trivy on PATH. Between the runs the stubs finish in the opposite order, and Trivy lists its two results (lines 81 and 74 of one file) in the opposite order. "On main" means the branch's test file run against 2e50cc2:

Test Branch On main
scanner completion order and Trivy result order do not change the report (JSON and HTML byte-identical) pass fails: sorted by scanner, file and line: expected ["semgrep:1","trivy:74","trivy:81"], got ["semgrep:1","trivy:81","trivy:74"]
compareFindings sorts all 720 orders of six findings with odd values (lines null, "1", "unknown", 2; files 10, "9") to one list pass not run on main (no compareFindings); at b98b966: expected 1, got 5
the other 13 determinism tests pass pass

The rehearsal's case with this branch: juice-shop at 266448a, run twice with SECGATE_TIMESTAMP=2026-01-01T00:00:00Z node secgate.js <juice-shop> --output-dir <dir>:

Run 1 Run 2
Result FAIL, exit 1, score 0, 11,034 findings (Trivy 10,726, Gitleaks 195, Semgrep 71, npm audit 42), 102 s the same, 28 s with warm caches
secgate-v7-report.json sha256 39df6c2a…ffb43a20 39df6c2a…ffb43a20
HTML report byte-identical to run 1
git status of the target clean clean

Gate

npm ci && npm run check:deps && npm run lint && npm test on Node 22.23.4 at c3d83c2, exit 0:

  • dependency guard passed, eslint clean;
  • engine 59, score 22, walk ok, smoke 55, schema 6, SARIF 11, trivy-image 9, no-lockfile 6, config 15, baseline 7, suppression 8, confidence 28, timeout ok, determinism 15, golden 8, action-exit ok; 0 failed.

Not changed

  • Dedup still keeps the first of two findings with the same (tool, signature, file, image, line).
  • The HTML report still groups findings by severity; inside a severity they now follow this order.
  • The other rehearsal findings are not part of this change.

@danieloceno danieloceno added the bug Something isn't working label Nov 16, 2026
@danieloceno

Copy link
Copy Markdown
Collaborator Author

Second-model review

One pass by a second model over the branch at b98b966.

  1. compareFindings is not a total order for odd location values (blocking), lib/scanners.mjs:260-264.
    • Problem: line and col were compared by subtraction, and the later keys were used only when the result was falsy. The scanner adapters pass locations through unchecked (lib/scanners.mjs:387-390 and 505-508). For one rule and file with lines null, "1" and "unknown" the comparator says null < "1", "1" < "unknown" and null > "unknown", so the sorted order still depends on the input order.
    • Proof: two otherwise identical Semgrep stub outputs with those three results in opposite orders, all other scanners off and SECGATE_TIMESTAMP fixed, gave different JSON reports (null,1,unknown against unknown,null,1).
    • Expected fix: compare line and col with an explicit order for missing values, finite numbers and anything else, keep the whole-finding tie-breaker, and add a test over input permutations.

Checked, no finding:

  • The sort runs as soon as every scanner has finished (secgate.js:496), before the baseline write and compare, score, status, the console list, and the JSON, HTML and SARIF reports. It changes no count, score, baseline match, verdict or exit code.
  • The new CLI test isolates PATH, reverses both the completion order and Trivy's result order, and removes its temporary folders. test/schema.mjs only stops reading the Gitleaks finding by its position.
  • README, docs/aggregation.md and the CHANGELOG describe the order correctly for normal, numeric scanner output.

Verdict: request changes (finding 1).

In plain English

SecGate could list the same problems in a different order on two runs of the same code, which made its promise of byte-identical results false. This change sorts the list, and the reviewer confirmed that the sorting happens before every report and does not change any score or verdict. The reviewer also found that the sorting rule broke down when a scanner sends an odd line number, such as a word instead of a number, so the order could still change in that rare case. That is fixed and tested in the next commit.

@danieloceno

Copy link
Copy Markdown
Collaborator Author

Fix for finding 1: c3d83c2

  • compareLocation orders line and col as: missing first, then finite numbers in numeric order, then any other value by its JSON text. compareText compares the string form of file, signature and image, so mixed value types there cannot make the order depend on the input order either. The whole-finding tie-breaker stays.
  • New test determinism: compareFindings sorts every input order the same way, whatever the value types: all 720 orders of six findings (lines null, "1", "unknown" and 2; files 10 and "9") must sort to one list. At b98b966 they sorted to 5 different lists (expected 1, got 5); at c3d83c2 to one.
  • The review's CLI reproduction (the three Semgrep results in opposite orders) now writes the same report for both orders.
  • Normal numeric output sorts as before, so the end-to-end test keeps its expected order.

Full gate at c3d83c2, Node 22.23.4, exit 0: npm ci, check:deps, lint, and npm test (engine 59, score 22, smoke 55, schema 6, SARIF 11, trivy-image 9, no-lockfile 6, config 15, baseline 7, suppression 8, confidence 28, determinism 15, golden 8; walk, timeout and action-exit ok; 0 failed). CI 10/10 green at c3d83c2.

@danieloceno
danieloceno merged commit 30ac483 into main Nov 16, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant