Repository navigation
A dedumped sample is not a dump, and an empty base address is not zero (#44) - #141
Conversation
89e02a8 to
40caf69
Compare
Issue #44 asks whether a filename containing "dedumped" should be treated as unmapped. The answer is yes, and the code was already doing the opposite: "dedumped" contains "dump", so request_filename_info classified it as a dump and submit_or_query_dropzone.html clicked #dumped on that answer. Since a de-dumped file has been *un*mapped, it normally carries no _0x... in its name, so the user landed in "Dumped" with an empty base address and no bitness. Submitting that prefilled form was a 500, in both submit paths: /analyze/query options=dumped base_addr="" -> 500 /data/submit options=dumped base_addr="" -> 500 ValueError: invalid literal for int() /data/submit options=dumped, no bitness -> 400 KeyError Reproducing that turned up a third one the issue does not mention: int(x, 16) accepts a leading minus, so a negative base address was accepted and forwarded to the backend as base_address=-4194304. Now rejected. Both fields are validated in params.py rather than inline, because the base address is parsed at two call sites in two blueprints and one helper keeps them from drifting. Neither defaults: guessing a base address for a memory dump is a worse failure than an error message, and the dropzone's own error handler reloads the page on a non-2xx - which is exactly what renders the flash - so flash + 400 is the mechanism that actually tells the user. The exclusion is the literal token rather than a (?<!de)dump lookbehind. The lookbehind also suppresses "widedump" and "sidedump", and anchoring it with \b does not help because _ is a word character in Python's re, which re-breaks "sample_dedumped" - the exact case this is about. The token's one false negative is a name carrying both "dedumped" and a real dump marker, which costs one radio-button click. The smda half of the issue belongs to that project, not this one.
base address the SMDA path never reads Two findings from a review of this change. `'dedumped' not in filename` is defeated by every separator and stem variant, so issue #44's own symptom survived them. Measured through `POST /data/request_filename_info` before the change: `sample_de-dumped.bin`, `sample_de_dumped.bin`, `sample_dedump.bin` and `sample_dedumping.bin` all came back `dump: True` with an empty base address - the dropzone clicks "Dumped", prefills nothing, and the submit is refused. `sample_DeDumped.bin` came out right only because `'dump' in filename` is *also* case-sensitive: two bugs cancelling. The rule is now: remove the de-dump markers, then ask what dump is left. DEDUMP_IN_FILENAME = re.compile(r"(?<![a-zA-Z])de[-_. ]?dump", re.IGNORECASE) DUMP_IN_FILENAME = re.compile(r"dump", re.IGNORECASE) `\b` cannot work here - `_` is a word character to `re`, so `sample_dedumped` is one token - and a strict token test would break `memdump`, which an existing test pins as a dump. So only the de-dump marker is anchored, and on its left only: "de" opens a de-dump where it starts a token and is the tail of the word before it where it does not, which is what keeps `widedump` and `sidedump` dumps. Subtracting rather than testing independently also fixes the false negative this PR flagged and left: `dump_dedumped_0x400000.bin` is a dump again. One deliberate behaviour change from master: `SAMPLE_DUMP.BIN` is now recognised. The exclusion has to be case-insensitive to catch `DEdumped`, and leaving the positive test case-sensitive would keep the cancelling asymmetry as a rule rather than an accident. `MEMDUMP.BIN` is a real spelling, and a false positive costs one radio click where the false negative costs filling the dump form by hand. Separately, `is_dump_or_smda` gated the base-address and bitness validation, but the SMDA path reads neither: `addReport(smda_report)` and `requestMatchesForSmdaReport(...)` take no such parameter - checked in the installed client, and the `else` branch in `data.submit` is unreachable because `SmdaReport.fromDict` raises rather than returning falsy. So a valid SMDA report was refused 400 on a field nobody reads, whenever the dropzone's header sniff missed. Gated on the dump alone now. That leaves the parse as the first thing an unreadable body reaches, so it gets its own guard. Not a drive-by: the old 400 was accidental cover with a misleading message, and the 500 behind it was already reachable on master whenever the address happened to be filled - which the dropzone does for a `.smda` drop. Five unreadable bodies across both routes now answer 400 and reach no backend call. 276 -> 315.
8b5110c to
2bd571c
Compare
Several routes parsed request input without handling a bad value, so a hand-made request ended in an unhandled exception: - cross_compare: a non-numeric samples or cache list (int()) - import: JSON nested deeper than json.load follows (RecursionError) - request_filename_info: a filename that is not a string, and a base_addr in the file header longer than int() converts - submit_or_query: an unknown form_type, for which the view returned nothing - the job page: a refresh or forward value that is not a number - /api/query/function and POST /api/samples: a body that is not a report, which SmdaReport.fromDict raised on, or nested too deep for get_json Each now answers 400, or reads the value as absent where it is only a display option. The same kind of parse in analyze.query and submit is left to #141, which already reworks both. The function comparison's edit distance mapped instructions to the characters chr(0x20) to chr(0xff) and raised past 223 distinct ones, so two large or unusual functions failed the comparison page. The symbols now run up the code points, skipping the surrogates; rapidfuzz compares them as fast. Such a pair now costs what any pair of that size costs in the block-pair loop, instead of failing early. test_the_api_blueprint_is_exempt took any 400 from POST /api/samples for the CSRF refusal; that route now answers {} with its own 400, so the test looks for the CSRF message instead.
Several routes parsed request input without handling a bad value, so a hand-made request ended in an unhandled exception: - cross_compare: a non-numeric samples or cache list (int()) - import: JSON nested deeper than json.load follows (RecursionError) - request_filename_info: a filename that is not a string, and a base_addr in the file header longer than int() converts - submit_or_query: an unknown form_type, for which the view returned nothing - the job page: a refresh or forward value that is not a number - /api/query/function and POST /api/samples: a body that is not a report, which SmdaReport.fromDict raised on, or nested too deep for get_json Each now answers 400, or reads the value as absent where it is only a display option. The same kind of parse in analyze.query and submit is left to #141, which already reworks both. The function comparison's edit distance mapped instructions to the characters chr(0x20) to chr(0xff) and raised past 223 distinct ones, so two large or unusual functions failed the comparison page. The symbols now run up the code points, skipping the surrogates; rapidfuzz compares them as fast. Such a pair now costs what any pair of that size costs in the block-pair loop, instead of failing early. test_the_api_blueprint_is_exempt took any 400 from POST /api/samples for the CSRF refusal; that route now answers {} with its own 400, so the test looks for the CSRF message instead.
|
Reviewed for the batch C integration (#298, #299). The server side is right: a malformed or negative base address is now a flashed 400 instead of a 500, the SMDA path correctly stops asking for a base address, and an unmapped submit still sends no 1. An uppercase dump name is detected, but its address isn't parsed. 2bd571c made 2. A suffixed de-dump counts as a fully pre-filled dump. After the second commit, Older than this PR, but worth folding in while you're here:
It also needs master merged in (#249). 🤖 Generated with Claude Code |
…y case A de-dump tool that appends "_dedumped" to the dump's own name leaves the dump marker and the base address in it. Dropping the de-dump marker and asking what was left took foo_dump_0x00400000_dedumped for a fully pre-filled dump, which would submit an unmapped file with is_dump and the dump's base address. Any de-dump marker now means not a dump. The dump marker became case-insensitive in 2bd571c, but the address after it was still matched as "_0x" only, so FOO_DUMP_0X00400000 was a dump with no base address and no bitness, and its submit was refused. The address is now read after "_0x" or "_0X". The address also takes 1 to 16 digits and must end there: fewer than 8 digits used not to parse at all, and a longer run was cut to its first 16 digits. The bitness is still read off the digit count.
McritClient.addBinarySample leaves any other bitness out of the request, and the server then disassembles the dump as 32 bit, so a 16 sent by hand was dropped without a word. It is now refused like any other bad value.
|
Thanks. All of it is in, with master merged in (8ca0d32, no conflicts). 1. Uppercase address. f26e227 reads the address after 2. Suffixed de-dump. f26e227 lets any de-dump marker win: Address length. Same commit: Bitness 16. 9e58fcc accepts only 32 and 64 ( Each guard was checked by taking it out: |
|
CI is red only in |
Closes #44. The issue's own note that this "might be relevant ... for other scripts, e.g. in smda" points at a different codebase; the MCRITweb side is what this settles.
Answers upstream issue #44 — and the answer turns out to be "yes, and the code currently does the opposite".
What's broken
request_filename_infoclassifies by substring, and"dedumped"contains"dump":table/submit_or_query_dropzone.html:54then clicks#dumpedon that answer. A de-dumped file has been unmapped, so it normally carries no_0x…in its name — which means the user lands in "Dumped" with an empty base address and no bitness. Submitting that prefilled form is a 500, in both submit paths.How to see it yourself
This branch's
tests/testUpload.py(111 tests, 82 of them new) on master's code (9e07da2) gives 52 failed, 59 passed; on this branch they all pass.Before
After
The
-0x400000 -> 202line is a third bug the reproduction turned up that the issue does not mention:int(x, 16)accepts a leading minus, so a negative base address was silently accepted and forwarded to the backend asrequestMatchesForMappedBinary(base_address=-4194304). It is now rejected.Why this fix
A de-dump marker anchored on its left.
DEDUMP_IN_FILENAME(mcritweb/views/params.py:203) is(?<[a-zA-Z])de[-_. ]?dump, case-insensitive. A plain(?<!de)dumpwould suppress anydumppreceded by the lettersde—widedump,sidedumpwould become "unmapped". Anchoring it with\bdoes not help, because_is a word character in Python'sre, so\bdemisses the exact case the issue is about (sample_dedumped). Requiring no letter beforedekeepswidedumpa dump and catchesde-dumped,de_dumped,.DEDUMPED.anddedump.Any de-dump marker wins.
parseIsDumpFromFilename(params.py:207) answers "dump" only for a name with a dump marker and no de-dump marker. A de-dump tool that appends_dedumpedto the dump's own name leaves both, plus the dump's base address (foo_dump_0x00400000_dedumped); taking that for a dump would submit an unmapped file withis_dump=1&base_addr=0x400000.Flash + 400, not a default.
int(request.form.get('base_addr') or '0', 16)would silently submit a memory dump at base 0, which is a worse failure than an error. And this is confirmed as the right shape by the client:myDropzone.on('error', function(file, response) { location.reload(); })— a non-2xx reloads the page, which is exactly what renders the flashed message. Soflash(...); return "", 400is not merely consistent with the"Please upload a file"path two lines above; it is the mechanism that actually shows the user the error. Before this, a 500 also reloaded, with nothing said.Validation in
params.py, not inline. The base address is parsed at two call sites in two blueprints; one helper keeps them from drifting, andparams.pyis explicitly the module for request-parameter parsing importable without Flask globals or the DB. It also keeps the footprint insidedata.py/analyze.pyto three small hunks, which matters — those two files are the most contended in the current batch.Range and sign checks.
re.fullmatch("(0[xX])?[0-9a-fA-F]+", …)then<= 0xFFFFFFFFFFFFFFFF. Rejects negatives, rejects wider-than-64-bit, and (unlike a plainint(…, 16)) rejects PEP 515 underscore separators and a leading+. Bitness is whitelisted to32and64(params.py:194) rather than anyint():McritClient.addBinarySamplein mcrit 1.13.0 leaves any other value out of the request, and the server then takes the dump as 32 bit.What I didn't do
requiredattribute on#base_addr. It would be inert: the form is never natively submitted —myDropzone.on('sending', …)callsserializeArray()and appends the fields by hand, so the browser runs no constraint validation. Worse, the field lives inside#extendedFields, which isdisplay:nonewhenever "Unmapped" is selected, and a hiddenrequiredcontrol is a well-known way to wedge native submission if the form ever becomes native. The server-side guard is the enforcement.Pre-existing, reported rather than shipped
request_filename_info500s on a non-stringfilename: thetry/exceptcovers only the extraction, andfilename.endswith(".smda")sits outside it.Not reachable from the real dropzone (
file.nameis always a string) and it needs a logged-in contributor, so it is a different bug from #44 and I kept it out of the diff. #283 adds theisinstanceguard.Risks, flagged honestly
int(x, 16)used to accept now 400:-0x400000(deliberate — it was reaching the backend),+0x400000, and0x1_0000. The latter two are Python-isms nobody types into a base-address field, and both are covered by named tests, but it is technically a narrowing rather than a pure bug fix.0x0is still accepted as an explicit user-typed base address. Rejecting a value someone typed on purpose is a different thing from defaulting to it — but if base 0 is always wrong here, that is a one-line change.dump_dedumped_0x400000.bin) is offered as unmapped, so a genuine dump with such a name costs the user one click on "Dumped" and the base address. The other way round would submit an unmapped file as a dump, silently.SAMPLE_DUMP.BINis offered as a dump, and the address after it is read after_0xor_0X(BASE_ADDR_IN_FILENAME,params.py:217), soSAMPLE_DUMP_0X00400000.BINis pre-filled as 32 bit at 0x400000.foo_dump7_0x400000now pre-fills 0x400000 (master needed at least 8 digits), and an 18-digit run is no address at all rather than its first 16 digits. The bitness is still read off the digit count: more than 8 is 64 bit (params.py:233).Rebase onto master (1cdc195)
Rebased from 98cfdc0. Master has since merged the filename parsing into one
parseBaseAddrAndBitnessFromFilename(#190), and the newparseIsDumpFromFilenamesits next to it. Master also no longer writes the query upload inanalyze.query, so that hunk now only guards the SMDA parse.install the test tools in CI and in make initdropped out, since master already carries it. The full suite passes with Chromium (1352 passed, 6 skipped; ruff clean), and a trial merge with #271 to #283 is clean. #283 leaves the base address, bitness and.smdaparsing inanalyze.queryanddata.submitto this PR.Review round (comment 6270204891)
Master (9e07da2) is merged in, with no conflicts. Then:
_0xor_0X; 1 to 16 digits with a right boundary. Tests:SAMPLE_DUMP_0X00400000.BIN,SAMPLE_DUMP_0X140000000.BIN,foo_dump7_0x400000, an 18-digit address, andfoo_dump_0x00400000_dedumpedwith two other both-marker names as unmapped.16joins the refused values.Each guard was checked by taking it out:
_0xonly fails the two_0Xcases, the old drop-and-ask-again rule fails the three both-marker names, no right boundary fails the 18-digit case,{8,16}failsfoo_dump7_0x400000, and accepting16fails its case.Full suite with Chromium on 9e58fcc: 1462 passed, 7 skipped;
ruff check .clean.Master (6564602, batch C) is merged in as well. The one conflict was the
paramsimport inmcritweb/views/analyze.py:11, where #174 addedsearch_pagenext to it; both imports are kept.