Skip to content

Bound the hash list of a cross compare and report what it ignored once - #280

Merged
danielplohmann merged 2 commits into
masterfrom
fix/hash-list-cross-compare
Oct 24, 2026
Merged

danielplohmann merged 2 commits into
masterfrom
fix/hash-list-cross-compare

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

/analyze/cross_compare_from_hash_list looks every pasted line up with getSampleBySha256, one backend request per line, with no limit and repeats included. For every line it ignores it flashes a message of its own. Flashes are stored in Flask's cookie session, so a list of mostly unknown hashes (say, sample hashes from a report, few of them in this corpus) outgrows what a browser keeps: in the test client, 100 random hashes made a 5692-byte session cookie, and Werkzeug warns that browsers may ignore anything past 4093. The page then shows none of the messages. The lookups themselves were measured in an earlier review at 7000 requests in 11.5 s for a 455 KB list.

Change. In cross_compare_from_hash_list (mcritweb/views/analyze.py:210-240):

  • Lines are stripped, blank lines are skipped, and valid hashes are lowercased and deduplicated in order. Every sample mcrit hashes itself is stored as a lowercase hexdigest() and the backend matches exactly, so an uppercase hash named no sample before; a repeated one selected the same sample twice.
  • The list is cut at MAX_HASH_LIST_LENGTH distinct hashes before any lookup (analyze.py:27, :225), with one warning saying how many were dropped. It is the same 250 as MAX_SELECTED_SAMPLES, the bound the unique blocks selection already has; the form label names it (mcritweb/templates/cross_compare_from_hash_list.html:25).
  • Malformed lines and unknown hashes are each reported in one warning that gives the count and names the first five (analyze.py:33, :221, :238). Flash messages are autoescaped in base.html, as before.

Verification. On the live testbed (mcrit 1.13.0), master on :5000 against this branch on a side instance, the same pasted list: three known hashes, the first one again in upper case and once more as is, 100 random hashes and one malformed line. curl as the client, which, like a browser, drops a cookie over 4096 bytes.

master (1cdc195) this branch
Set-Cookie header of the POST 5837 bytes 644 bytes
warnings shown on the page it redirects to 0 2
samples in the redirect 0,1,2,0 0,1,2
backend requests (counting proxy, branch only) 103

Measured on 1734122; the commit was then amended only to deduplicate with dict.fromkeys instead of a list membership test, which was quadratic before the cap applied, and to write the cap out in the tests. The outputs are the same. Only lookups were sent, and the corpus check afterwards reports it unchanged.

Tests. tests/testCrossCompareHashList.py, new:

  • 100 unknown hashes keep the session cookie under 4093 bytes, and the one warning names the first hash, not the last.
  • Malformed lines are reported in one warning that does not count the blank line.
  • A hash given twice and once in upper case is looked up once, in lowercase, and selects its sample.
  • A list 30 over the cap is cut before any lookup.
  • The form names the cap.
  • The cap the tests write out (250) is the view's MAX_HASH_LIST_LENGTH.

All six fail with master's analyze.py and template: the first five on behaviour, the last because master has no such constant.

Relation to open PRs. A trial merge (git merge-tree, against this head) with #219, #244, #253 and #271 to #279 is clean. #174 edits the search_page() call of this route and already conflicts with master in analyze.py; the set of conflicting files is the same with and without this branch.

Left out. The findLoops cost on large graphs (cfg_explorer_detector.py), found in the same review, is not touched here: #270 changes that view, so a change there should follow it.

cross_compare_from_hash_list looked up every pasted line, repeats
included, with one backend request each, and flashed a message per line
it ignored. Flashes live in the cookie session, which passed the 4093
bytes a browser keeps at about 70 unknown hashes, so a long list ended
with none of the messages shown.

Lines are now deduplicated and lowercased (the corpus stores lowercase
hashes and the backend matches exactly), the list is cut at 250 hashes
like the unique blocks selection, and the ignored lines are reported in
one warning each for malformed and unknown hashes, naming the first
five.
@danielplohmann
danielplohmann merged commit dfb951d into master Oct 24, 2026
8 checks passed
danielplohmann added a commit that referenced this pull request Oct 24, 2026
@r0ny123
r0ny123 deleted the fix/hash-list-cross-compare branch October 30, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants