Skip to content

A dedumped sample is not a dump, and an empty base address is not zero (#44) - #141

Merged
danielplohmann merged 7 commits into
familiary:masterfrom
r0ny123:fix/44-dedumped-is-not-a-dump
Oct 29, 2026
Merged

danielplohmann merged 7 commits into
familiary:masterfrom
r0ny123:fix/44-dedumped-is-not-a-dump

Conversation

@r0ny123

@r0ny123 r0ny123 commented Aug 30, 2026 •

Copy link
Copy Markdown
Collaborator

Mirrors r0ny123/mcritweb#39 from a triage series run against df53db9. Bare #N in the text below are upstream issue numbers; links point at the sibling PRs of the series.

Closes #44. The issue's own note that this "might be relevant ... for other scripts, e.g. in smda" points at a different codebase; the MCRITweb side is what this settles.

Answers upstream issue #44 — and the answer turns out to be "yes, and the code currently does the opposite".

What's broken

request_filename_info classifies by substring, and "dedumped" contains "dump":

elif 'dump' in filename:           # data.py:1443 on master 1cdc195
    result['dump'] = True

table/submit_or_query_dropzone.html:54 then clicks #dumped on that answer. A de-dumped file has been unmapped, so it normally carries no _0x… in its name — which means the user lands in "Dumped" with an empty base address and no bitness. Submitting that prefilled form is a 500, in both submit paths.

How to see it yourself

git clone https://github.com/fkie-cad/mcritweb && cd mcritweb
git fetch https://github.com/r0ny123/mcritweb fix/44-dedumped-is-not-a-dump && git checkout FETCH_HEAD
make init
.venv/bin/python -m pytest tests/testUpload.py -q

This branch's tests/testUpload.py (111 tests, 82 of them new) on master's code (9e07da2) gives 52 failed, 59 passed; on this branch they all pass.

Before

== classification ==
  sample_dedumped.bin              {"base_addr": "", "bitness": null, "dump": true}
  sample_dedumped_0x00400000.bin   {"base_addr": "0x400000", "bitness": 32, "dump": true}
  dedumped_thing                   {"base_addr": "", "bitness": null, "dump": true}
  notepad.dumped                   {"base_addr": "", "bitness": null, "dump": true}
  malware_dump_0x140000000.bin     {"base_addr": "0x140000000", "bitness": 64, "dump": true}
  sample.exe                       {"dump": false}
== submit paths ==
  /analyze/query  {'options': 'dumped', 'base_addr': ''}                             -> 500
  /analyze/query  {'options': 'dumped', 'base_addr': 'zzz'}                          -> 500
  /analyze/query  {'options': 'dumped', 'base_addr': '-0x400000'}                    -> 202
  /data/submit    {'options': 'dumped', 'base_addr': '', 'bitness': '32'}            -> 500
  /data/submit    {'options': 'dumped', 'base_addr': '0x400000', 'bitness': ''}      -> 500
  /data/submit    {'options': 'dumped', 'base_addr': '0x400000'}                     -> 400
  /data/submit    {'options': 'dumped', 'base_addr': '0x140000000', 'bitness': '64'} -> 202

After

== classification ==
  sample_dedumped.bin              {"dump": false}
  sample_dedumped_0x00400000.bin   {"dump": false}
  dedumped_thing                   {"dump": false}
  notepad.dumped                   {"base_addr": "", "bitness": null, "dump": true}
  malware_dump_0x140000000.bin     {"base_addr": "0x140000000", "bitness": 64, "dump": true}
  sample.exe                       {"dump": false}
== submit paths ==
  /analyze/query  {'options': 'dumped', 'base_addr': ''}                             -> 400
  /analyze/query  {'options': 'dumped', 'base_addr': 'zzz'}                          -> 400
  /analyze/query  {'options': 'dumped', 'base_addr': '-0x400000'}                    -> 400
  /data/submit    {'options': 'dumped', 'base_addr': '', 'bitness': '32'}            -> 400
  /data/submit    {'options': 'dumped', 'base_addr': '0x400000', 'bitness': ''}      -> 400
  /data/submit    {'options': 'dumped', 'base_addr': '0x400000'}                     -> 400
  /data/submit    {'options': 'dumped', 'base_addr': '0x140000000', 'bitness': '64'} -> 202

The -0x400000 -> 202 line is a third bug the reproduction turned up that the issue does not mention: int(x, 16) accepts a leading minus, so a negative base address was silently accepted and forwarded to the backend as requestMatchesForMappedBinary(base_address=-4194304). It is now rejected.

239 on master -> 276  (+37 in tests/testUpload.py)
276 passed, 1 warning in 42.64s
All checks passed!            # ruff

Why this fix

A de-dump marker anchored on its left. DEDUMP_IN_FILENAME (mcritweb/views/params.py:203) is (?<[a-zA-Z])de[-_. ]?dump, case-insensitive. A plain (?<!de)dump would suppress any dump preceded by the letters de — widedump, sidedump would become "unmapped". Anchoring it with \b does not help, because _ is a word character in Python's re, so \bde misses the exact case the issue is about (sample_dedumped). Requiring no letter before de keeps widedump a dump and catches de-dumped, de_dumped, .DEDUMPED. and dedump.

Any de-dump marker wins. parseIsDumpFromFilename (params.py:207) answers "dump" only for a name with a dump marker and no de-dump marker. A de-dump tool that appends _dedumped to the dump's own name leaves both, plus the dump's base address (foo_dump_0x00400000_dedumped); taking that for a dump would submit an unmapped file with is_dump=1&base_addr=0x400000.

Flash + 400, not a default. int(request.form.get('base_addr') or '0', 16) would silently submit a memory dump at base 0, which is a worse failure than an error. And this is confirmed as the right shape by the client: myDropzone.on('error', function(file, response) { location.reload(); }) — a non-2xx reloads the page, which is exactly what renders the flashed message. So flash(...); return "", 400 is not merely consistent with the "Please upload a file" path two lines above; it is the mechanism that actually shows the user the error. Before this, a 500 also reloaded, with nothing said.

Validation in params.py, not inline. The base address is parsed at two call sites in two blueprints; one helper keeps them from drifting, and params.py is explicitly the module for request-parameter parsing importable without Flask globals or the DB. It also keeps the footprint inside data.py / analyze.py to three small hunks, which matters — those two files are the most contended in the current batch.

Range and sign checks. re.fullmatch("(0[xX])?[0-9a-fA-F]+", …) then <= 0xFFFFFFFFFFFFFFFF. Rejects negatives, rejects wider-than-64-bit, and (unlike a plain int(…, 16)) rejects PEP 515 underscore separators and a leading +. Bitness is whitelisted to 32 and 64 (params.py:194) rather than any int(): McritClient.addBinarySample in mcrit 1.13.0 leaves any other value out of the request, and the server then takes the dump as 32 bit.

What I didn't do

  • The smda-side half of the issue is not this repository's. The submitter's note that this "might be relevant … for other scripts, e.g. in smda" points at a different codebase. Left alone.
  • No required attribute on #base_addr. It would be inert: the form is never natively submitted — myDropzone.on('sending', …) calls serializeArray() and appends the fields by hand, so the browser runs no constraint validation. Worse, the field lives inside #extendedFields, which is display:none whenever "Unmapped" is selected, and a hidden required control is a well-known way to wedge native submission if the form ever becomes native. The server-side guard is the enforcement.
  • No error text in the 400 body — the dropzone discards the body and reloads, so the flash is what the reader sees.

Pre-existing, reported rather than shipped

request_filename_info 500s on a non-string filename: the try/except covers only the extraction, and filename.endswith(".smda") sits outside it.

{"filename": 123,  "file_header": ""}  -> 500
{"filename": null, "file_header": ""}  -> 500

Not reachable from the real dropzone (file.name is always a string) and it needs a logged-in contributor, so it is a different bug from #44 and I kept it out of the diff. #283 adds the isinstance guard.

Risks, flagged honestly

  • A small behavioural narrowing. Three values int(x, 16) used to accept now 400: -0x400000 (deliberate — it was reaching the backend), +0x400000, and 0x1_0000. The latter two are Python-isms nobody types into a base-address field, and both are covered by named tests, but it is technically a narrowing rather than a pure bug fix.
  • 0x0 is still accepted as an explicit user-typed base address. Rejecting a value someone typed on purpose is a different thing from defaulting to it — but if base 0 is always wrong here, that is a one-line change.
  • A filename containing both a de-dump and a dump marker (dump_dedumped_0x400000.bin) is offered as unmapped, so a genuine dump with such a name costs the user one click on "Dumped" and the base address. The other way round would submit an unmapped file as a dump, silently.
  • Dump detection is case-insensitive now, so SAMPLE_DUMP.BIN is offered as a dump, and the address after it is read after _0x or _0X (BASE_ADDR_IN_FILENAME, params.py:217), so SAMPLE_DUMP_0X00400000.BIN is pre-filled as 32 bit at 0x400000.
  • The address in a name takes 1 to 16 digits, and must end there. foo_dump7_0x400000 now pre-fills 0x400000 (master needed at least 8 digits), and an 18-digit run is no address at all rather than its first 16 digits. The bitness is still read off the digit count: more than 8 is 64 bit (params.py:233).

Rebase onto master (1cdc195)

Rebased from 98cfdc0. Master has since merged the filename parsing into one parseBaseAddrAndBitnessFromFilename (#190), and the new parseIsDumpFromFilename sits next to it. Master also no longer writes the query upload in analyze.query, so that hunk now only guards the SMDA parse. install the test tools in CI and in make init dropped out, since master already carries it. The full suite passes with Chromium (1352 passed, 6 skipped; ruff clean), and a trial merge with #271 to #283 is clean. #283 leaves the base address, bitness and .smda parsing in analyze.query and data.submit to this PR.

Review round (comment 6270204891)

Master (9e07da2) is merged in, with no conflicts. Then:

  • f26e227: any de-dump marker wins; the address is read after _0x or _0X; 1 to 16 digits with a right boundary. Tests: SAMPLE_DUMP_0X00400000.BIN, SAMPLE_DUMP_0X140000000.BIN, foo_dump7_0x400000, an 18-digit address, and foo_dump_0x00400000_dedumped with two other both-marker names as unmapped.
  • 9e58fcc: bitness only 32 or 64; 16 joins the refused values.

Each guard was checked by taking it out: _0x only fails the two _0X cases, the old drop-and-ask-again rule fails the three both-marker names, no right boundary fails the 18-digit case, {8,16} fails foo_dump7_0x400000, and accepting 16 fails its case.

Full suite with Chromium on 9e58fcc: 1462 passed, 7 skipped; ruff check . clean.

Master (6564602, batch C) is merged in as well. The one conflict was the params import in mcritweb/views/analyze.py:11, where #174 added search_page next to it; both imports are kept.

Issue #44 asks whether a filename containing "dedumped" should be treated as
unmapped. The answer is yes, and the code was already doing the opposite:
"dedumped" contains "dump", so request_filename_info classified it as a dump
and submit_or_query_dropzone.html clicked #dumped on that answer.

Since a de-dumped file has been *un*mapped, it normally carries no _0x... in
its name, so the user landed in "Dumped" with an empty base address and no
bitness. Submitting that prefilled form was a 500, in both submit paths:

    /analyze/query  options=dumped base_addr=""  -> 500
    /data/submit    options=dumped base_addr=""  -> 500   ValueError: invalid
                                                          literal for int()
    /data/submit    options=dumped, no bitness   -> 400   KeyError

Reproducing that turned up a third one the issue does not mention: int(x, 16)
accepts a leading minus, so a negative base address was accepted and forwarded
to the backend as base_address=-4194304. Now rejected.

Both fields are validated in params.py rather than inline, because the base
address is parsed at two call sites in two blueprints and one helper keeps
them from drifting. Neither defaults: guessing a base address for a memory
dump is a worse failure than an error message, and the dropzone's own error
handler reloads the page on a non-2xx - which is exactly what renders the
flash - so flash + 400 is the mechanism that actually tells the user.

The exclusion is the literal token rather than a (?<!de)dump lookbehind. The
lookbehind also suppresses "widedump" and "sidedump", and anchoring it with \b
does not help because _ is a word character in Python's re, which re-breaks
"sample_dedumped" - the exact case this is about. The token's one false
negative is a name carrying both "dedumped" and a real dump marker, which
costs one radio-button click.

The smda half of the issue belongs to that project, not this one.
base address the SMDA path never reads

Two findings from a review of this change.

`'dedumped' not in filename` is defeated by every separator and stem variant,
so issue #44's own symptom survived them. Measured through
`POST /data/request_filename_info` before the change: `sample_de-dumped.bin`,
`sample_de_dumped.bin`, `sample_dedump.bin` and `sample_dedumping.bin` all came
back `dump: True` with an empty base address - the dropzone clicks "Dumped",
prefills nothing, and the submit is refused. `sample_DeDumped.bin` came out
right only because `'dump' in filename` is *also* case-sensitive: two bugs
cancelling.

The rule is now: remove the de-dump markers, then ask what dump is left.

    DEDUMP_IN_FILENAME = re.compile(r"(?<![a-zA-Z])de[-_. ]?dump", re.IGNORECASE)
    DUMP_IN_FILENAME = re.compile(r"dump", re.IGNORECASE)

`\b` cannot work here - `_` is a word character to `re`, so `sample_dedumped` is
one token - and a strict token test would break `memdump`, which an existing test
pins as a dump. So only the de-dump marker is anchored, and on its left only:
"de" opens a de-dump where it starts a token and is the tail of the word before
it where it does not, which is what keeps `widedump` and `sidedump` dumps.
Subtracting rather than testing independently also fixes the false negative this
PR flagged and left: `dump_dedumped_0x400000.bin` is a dump again.

One deliberate behaviour change from master: `SAMPLE_DUMP.BIN` is now recognised.
The exclusion has to be case-insensitive to catch `DEdumped`, and leaving the
positive test case-sensitive would keep the cancelling asymmetry as a rule rather
than an accident. `MEMDUMP.BIN` is a real spelling, and a false positive costs one
radio click where the false negative costs filling the dump form by hand.

Separately, `is_dump_or_smda` gated the base-address and bitness validation, but
the SMDA path reads neither: `addReport(smda_report)` and
`requestMatchesForSmdaReport(...)` take no such parameter - checked in the
installed client, and the `else` branch in `data.submit` is unreachable because
`SmdaReport.fromDict` raises rather than returning falsy. So a valid SMDA report
was refused 400 on a field nobody reads, whenever the dropzone's header sniff
missed. Gated on the dump alone now.

That leaves the parse as the first thing an unreadable body reaches, so it gets
its own guard. Not a drive-by: the old 400 was accidental cover with a misleading
message, and the 500 behind it was already reachable on master whenever the
address happened to be filled - which the dropzone does for a `.smda` drop. Five
unreadable bodies across both routes now answer 400 and reach no backend call.

276 -> 315.
@r0ny123
r0ny123 force-pushed the fix/44-dedumped-is-not-a-dump branch from 8b5110c to 2bd571c Compare October 9, 2026 12:18
r0ny123 added a commit that referenced this pull request Oct 13, 2026
Several routes parsed request input without handling a bad value, so a
hand-made request ended in an unhandled exception:
- cross_compare: a non-numeric samples or cache list (int())
- import: JSON nested deeper than json.load follows (RecursionError)
- request_filename_info: a filename that is not a string, and a
  base_addr in the file header longer than int() converts
- submit_or_query: an unknown form_type, for which the view returned
  nothing
- the job page: a refresh or forward value that is not a number
- /api/query/function and POST /api/samples: a body that is not a
  report, which SmdaReport.fromDict raised on, or nested too deep for
  get_json
Each now answers 400, or reads the value as absent where it is only a
display option. The same kind of parse in analyze.query and submit is
left to #141, which already reworks both.

The function comparison's edit distance mapped instructions to the
characters chr(0x20) to chr(0xff) and raised past 223 distinct ones, so
two large or unusual functions failed the comparison page. The symbols
now run up the code points, skipping the surrogates; rapidfuzz compares
them as fast. Such a pair now costs what any pair of that size costs
in the block-pair loop, instead of failing early.

test_the_api_blueprint_is_exempt took any 400 from POST /api/samples
for the CSRF refusal; that route now answers {} with its own 400, so
the test looks for the CSRF message instead.
r0ny123 added a commit that referenced this pull request Oct 13, 2026
Several routes parsed request input without handling a bad value, so a
hand-made request ended in an unhandled exception:
- cross_compare: a non-numeric samples or cache list (int())
- import: JSON nested deeper than json.load follows (RecursionError)
- request_filename_info: a filename that is not a string, and a
  base_addr in the file header longer than int() converts
- submit_or_query: an unknown form_type, for which the view returned
  nothing
- the job page: a refresh or forward value that is not a number
- /api/query/function and POST /api/samples: a body that is not a
  report, which SmdaReport.fromDict raised on, or nested too deep for
  get_json
Each now answers 400, or reads the value as absent where it is only a
display option. The same kind of parse in analyze.query and submit is
left to #141, which already reworks both.

The function comparison's edit distance mapped instructions to the
characters chr(0x20) to chr(0xff) and raised past 223 distinct ones, so
two large or unusual functions failed the comparison page. The symbols
now run up the code points, skipping the surrogates; rapidfuzz compares
them as fast. Such a pair now costs what any pair of that size costs
in the block-pair loop, instead of failing early.

test_the_api_blueprint_is_exempt took any 400 from POST /api/samples
for the CSRF refusal; that route now answers {} with its own 400, so
the test looks for the CSRF message instead.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Reviewed for the batch C integration (#298, #299). The server side is right: a malformed or negative base address is now a flashed 400 instead of a 500, the SMDA path correctly stops asking for a base address, and an unmapped submit still sends no base_addr. Two things in the filename parsing keep it out of the batch for now.

1. An uppercase dump name is detected, but its address isn't parsed. 2bd571c made DUMP_IN_FILENAME case-insensitive (params.py:204), while the address regex _0x(?P<base_addr>[0-9a-fA-F]{8,16}) (:219) is still case-sensitive. FOO_DUMP_0X00400000 therefore gives dump=True with no base and no bitness. The form selects "Dumped" with an empty base address, and the submit answers 400: #44's symptom, coming back through the new case rule. _0[xX] (or re.IGNORECASE) fixes it, with a SAMPLE_DUMP_0X00400000.BIN test.

2. A suffixed de-dump counts as a fully pre-filled dump. After the second commit, foo_dump_0x00400000_dedumped gives dump=True, 0x400000, 32. The first commit classified that name as not a dump, and the commit message says the flip is deliberate ("keeps the dump such a name still claims"). But a de-dump tool appending _dedumped to the dump's own name is the most likely real spelling. In that case the form is completely filled, and the backend receives is_dump=1&base_addr=0x400000 for an unmapped PE, which disassembles silently wrong. I'd let any de-dump marker win (not DEDUMP.search(name) and DUMP.search(name)), as the first commit did.

Older than this PR, but worth folding in while you're here:

  • An address shorter than 8 digits isn't parsed (foo_dump7_0x400000), and the regex has no right boundary, so an 18-digit address is silently cut to 16 digits. {1,16}(?![0-9a-fA-F]) handles both, with bitness decided separately.
  • parse_bitness_form_param accepts 16. McritClient.addBinarySample drops anything that isn't 32 or 64, and the server then defaults to 32. Accept only 32 and 64.

It also needs master merged in (#249).

🤖 Generated with Claude Code

https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG

…y case

A de-dump tool that appends "_dedumped" to the dump's own name leaves the
dump marker and the base address in it. Dropping the de-dump marker and
asking what was left took foo_dump_0x00400000_dedumped for a fully
pre-filled dump, which would submit an unmapped file with is_dump and the
dump's base address. Any de-dump marker now means not a dump.

The dump marker became case-insensitive in 2bd571c, but the
address after it was still matched as "_0x" only, so FOO_DUMP_0X00400000
was a dump with no base address and no bitness, and its submit was
refused. The address is now read after "_0x" or "_0X".

The address also takes 1 to 16 digits and must end there: fewer than 8
digits used not to parse at all, and a longer run was cut to its first 16
digits. The bitness is still read off the digit count.
McritClient.addBinarySample leaves any other bitness out of the request,
and the server then disassembles the dump as 32 bit, so a 16 sent by hand
was dropped without a word. It is now refused like any other bad value.
@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks. All of it is in, with master merged in (8ca0d32, no conflicts).

1. Uppercase address. f26e227 reads the address after _0x or _0X (BASE_ADDR_IN_FILENAME, mcritweb/views/params.py:217). SAMPLE_DUMP_0X00400000.BIN now gives dump=True, 0x400000, 32, and SAMPLE_DUMP_0X140000000.BIN gives 64 bit; both are tests.

2. Suffixed de-dump. f26e227 lets any de-dump marker win: not DEDUMP_IN_FILENAME.search(name) and DUMP_IN_FILENAME.search(name) (params.py:207). foo_dump_0x00400000_dedumped, dump_dedumped_0x00400000.bin and dump_dedumped_0x400000.bin are now offered as unmapped; the last two used to be tests for the opposite.

Address length. Same commit: _0[xX]([0-9a-fA-F]{1,16})(?![0-9a-fA-F]). foo_dump7_0x400000 pre-fills 0x400000 as 32 bit, and an 18-digit run is no address at all. The bitness is still read off the digit count, more than 8 being 64 bit (params.py:233).

Bitness 16. 9e58fcc accepts only 32 and 64 (params.py:194); 16 joins the refused values in test_a_dump_submit_without_a_usable_bitness_is_refused. Checked against mcrit 1.13.0: addBinarySample sends the bitness only when it is 32 or 64, and the server then defaults to 32.

Each guard was checked by taking it out: _0x only fails the two _0X cases, the old rule fails the three both-marker names, no right boundary fails the 18-digit case, {8,16} fails foo_dump7_0x400000, and accepting 16 fails its case. Full suite with Chromium: 1462 passed, 7 skipped; ruff clean. The PR description is updated to match.

@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI is red only in tests/testSubmitMetadata.py: 8 failed, 1437 passed on 9e58fcc. CI now installs mcrit 1.16.0, whose McritClient uses its own requests.Session subclass, and the test stub doesn't provide it. #297 fixes that. I left the fix out of this PR so the two don't overlap. Once #297 is merged, a rerun here should be green.

@danielplohmann
danielplohmann merged commit 85e6aa5 into familiary:master Oct 29, 2026
4 checks passed
danielplohmann added a commit that referenced this pull request Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider a sample to be unmapped if it contains "dedumped"?

2 participants