Repository navigation
Integrate batch C2 (UI): #132 #147 #148 #214 #159 #213 #226 #145 #229 #142 #115 #174 #212 (stacked on #298) - #299
Merged
Merged
Conversation
The search half of the issue was half-built and inert: the form in jobs.html sat inside a Jinja comment, so there was no search box at all, and data.jobs still read request.form['Search'] into a `query` it passed to the template and never used for anything. McritClient.getQueueData takes a `filter` string, which the backend applies as a substring test against each job's parameters, so the term now has somewhere to go. The box is back, as a GET rather than a POST: a search is a read, and in the URL it is linkable, survives a refresh, and is carried by the pagination links, which build from request.args. The route is GET-only again as a result - the POST branch also raised BadRequestKeyError for any POST without the field. The category, state and sort order ride along as hidden fields, so searching narrows the list you were looking at instead of resetting it. The sort half of the issue was already done, in an earlier release. One test case disappears with this: testCsrf's form scanner was matching the commented-out POST form. There is no POST form in jobs.html any more.
Every unit job on every Python version fails with:
/opt/hostedtoolcache/Python/3.11.16/x64/bin/python: No module named pytest
The Makefile said "pytest and pytest-cov arrive with mcrit, so requirements.txt
covers both". mcrit declares them under its `dev` extra:
$ python -c "import importlib.metadata as m; print(m.requires('mcrit'))"
... 'pytest; extra == "dev"', 'pytest-cov; extra == "dev"' ...
so a plain `pip install -r requirements.txt` does not bring them in, and the
suite has not actually run in CI since mcrit moved them there. Ruff is green
throughout, which is why this looked like a working pipeline.
Installed in the workflow rather than added to requirements.txt: neither is a
runtime dependency and a deployment should not get them. Pinned, for the same
reason the lint job pins ruff. The two stale comments that said otherwise are
corrected.
(cherry picked from commit 8268666)
work/harness/cookies.txt is a cookie jar written by a local debugging harness - a Flask session signed with a secret hardcoded in that harness, against a throwaway SQLite database. Not a credential for anything, but it has no business in this change. It arrived with the cherry-picked CI fix.
…at once
Review caught that passing the search term to getQueueData's own `filter` is
not just imprecise, it makes the feature worse than absent. mcrit slices the
requested page and filters what survives:
return [job._data for job in self.queue.get_jobs(start_index, limit,
method, state, ascending) if filter in job.parameters]
so `start=0, limit=25, filter=x` answers "the matches among jobs 0-24", not
"the first 25 matches". With 100 jobs and the only hit at number 60, page 1
of the search rendered empty and offered four pages counted from the
*unfiltered* total - the match was reachable only by guessing which page it
was on, and page 1 read as "no such job". I noticed the ordering, wrote a
comment about it, and shipped it anyway. That was wrong.
A search now fetches the category unpaged and filters here, so the count
driving pagination is the count of rows actually shown. Browsing without a
search term still uses the paged call and is unchanged.
Two things fall out of that. The term no longer reaches the backend at all,
which also removes it from a URL McritClient builds by f-string with no
encoding - a `#` in a term used to truncate the query and silently drop the
state filter, and an `&state=failed` used to inject a second state parameter.
And the filter is now case-insensitive, because doing it here means choosing,
and a search box that misses EVIL.EXE for evil.exe is a surprise.
The empty state is search-aware: "No jobs available. Click here to create
your first job" is the wrong thing to say to someone whose search missed on
a full queue, and it reads as "there are no jobs", which is false.
The cost is that a search fetches the whole category rather than a page.
That is the price of a correct answer; the alternative is a filter in the
backend that runs before the slice, which is an mcrit change.
tests/testJobSearch.py now carries PagedQueue, a fake with mcrit's real
slice-then-filter semantics. The corpus fake ignores start, limit and filter
and returns the whole queue, which is why nothing caught this. 7 of its 14
tests fail against the previous implementation.
Re-review of the previous fix: `(job.parameters or "")` guards against None,
but Job.parameters never returns None - it returns "" or raises. It calls
json.loads on payload["params"] and then .items() on the result, so a
document whose params is malformed, `null`, or a JSON array raises
JSONDecodeError, AttributeError or TypeError.
That matters because filtering the whole category means the search now
touches every job in it. With one corrupt job at index 80 of a hundred:
before this branch: ?Search=x 200 ?p=4 500 (browse)
with the filter: ?Search=x 500 every page of the search
So the fix turned a one-page browse failure into a whole-category search
failure. The `or ""` shows I had considered None and not the raise.
A job whose parameters cannot be read cannot contain the search term either,
so leaving it out is not a workaround - it is the right answer. The read is
logged so the corrupt document is still visible to an operator.
The pre-existing browse failure is untouched: that one dies in the template,
which reads the same property, and fixing it is a different change.
`init` still ran `pip install -r requirements.txt` alone, so a fresh checkout that follows the README lands without pytest and `make test` fails with "No module named pytest" - the same failure the CI change on this branch fixes for the runner, left unfixed for a contributor. pytest and pytest-cov are not runtime dependencies and are not in requirements.txt; mcrit declares them under its `dev` extra, which nothing here requests, so they do not arrive transitively either. Install them from `init`, pinned to the versions CI uses, and say so once above the target rather than in a comment further down that describes a step that never ran. Reported by Codex review.
The job list picks a tab by walking Job.method_types["all"] and taking the first category the live queue statistics mention. That choice appears nowhere in the URL - and the URL is the one thing a browser replays for a refresh or a step back in history, so neither can reproduce the page. The tab also moves on its own as soon as the queue changes underneath it. When the tab was chosen implicitly, redirect to the URL that names it, keeping the rest of the query. Every job-list history entry then carries its own tab, and F5 lands where it was. 'active' is now validated before it is used to index the statistics, which it was not: an invented category and a real-but-unqueued one both answered 500, and the second is a link this very view renders. The accepted set is method_types["all"], plus whatever the queue reports, plus the two maintenance categories the menu below links to that mcrit's own method_types omits. statistics.get() for the pagination count, so a category with no jobs renders an empty tab instead of a KeyError. The redirect builds its query with urlencode rather than passing the user's args to url_for as keyword arguments: the names url_for reserves are user input here, and ?endpoint=x is otherwise a TypeError. A loop is impossible - the default is drawn from the static method_types["all"], so it validates on the next request. Considered and rejected: a history.replaceState shim in the template, which is JS-dependent and leaves the server's own choice non-deterministic, and remembering the tab in the session, which makes the URL less authoritative still and breaks two job lists open at once. Refs #36.
`init` still ran `pip install -r requirements.txt` alone, so a fresh checkout that follows the README lands without pytest and `make test` fails with "No module named pytest" - the same failure the CI change on this branch fixes for the runner, left unfixed for a contributor. pytest and pytest-cov are not runtime dependencies and are not in requirements.txt; mcrit declares them under its `dev` extra, which nothing here requests, so they do not arrive transitively either. Install them from `init`, pinned to the versions CI uses, and say so once above the target rather than in a comment further down that describes a step that never ran. Reported by Codex review.
A match report carries one job-wide `match_aggregation`, computed by the backend. `MatchingResult.fromDict` copies it through verbatim and no filter in `applyFilterValues` touches it, so a result page narrowed to one family stated the numbers of the whole job. On the win.dridex view of the captured report that meant claiming 756 functions / 151654 bytes where the honest answer is 4 / 249. Issue #38. Four of the five fields are counts over the very function matches the report already contains, so aggregating the filtered list describes exactly what is on screen and needs nothing from the backend. That is what makes this presentation rather than a second opinion on the matching, which this repository does not get to have - and the warrant is that the recomputation reproduces the backend's own numbers exactly when nothing is filtered, for every captured report, asserted in the suite so it fails loudly if it ever stops holding. num_self_matches cannot be recomputed: _summarizeMatches drops matches of the sample against itself from the function list before serializing it. It is carried through job-wide and labelled "(whole job)" on a filtered page, rather than passing a job-wide number off as a filtered one. Dropping the row instead would delete real information on exactly the pages where someone is drilling in, and make the table's row set depend on the query string. is_filtered is the length of the filtered list against the full one, so a score or library filter narrows the table too - those left it equally wrong before. Two things worth recording because they were checked rather than assumed: the report's per-match flags exactly reconstruct membership in the backend's pichash/minhash sets, including under PICHASH_IMPLIES_MINHASH_MATCH, so the split is faithful; and bytes_matched sums in a different order from the backend's set, which is exact rather than merely close because all 2350 num_bytes values across the three reports are whole numbers far below 2**53.
pytest and ruff used to arrive as transitive dependencies of mcrit and no longer do, so a clean checkout could not run `make test` or `make lint`. Install them explicitly in both the workflow and the target that claims to set the environment up.
The four score cells on every match result page rendered the percentage with "%3d", which truncates toward zero rather than rounding: a sample scoring 85.88 showed as 85, a whole point below the 85.88 the tooltip on the same cell reports, and 0.76 showed as 0 beside a neighbour at 1.04 that had barely outscored it. That display gap is the shape of the "too far from the expected value" in #7 - the scores themselves come off MatchedSampleEntry unchanged and are the backend's to compute, not ours. "%3.0f" keeps the column width and the right-alignment, matches the "%5.2f" the tooltip beside it already uses, and rounds to nearest. Applied to all four score columns (direct, direct_nonlib, frequency, frequency_nonlib) in all four result templates so the columns stay comparable to each other. Refs #7
…k twice Issue #68. Four separate costs, measured on the offline corpus. The diagram dominated the first view of every result page. create_match_diagram ran synchronously at four points before render_template, and dragged in a getFunctionsBySampleId round trip with it. The image already had its own route and its own <img> request, so the work belongs there: /data/result/<sample> 102.05 ms -> 23.55 ms -77% /data/result/<sample>?famid=1 93.53 ms -> 24.30 ms -74% /data/result/<sample>?samid=1 84.95 ms -> 16.09 ms -81% /data/result/<sample>?funid=0 94.61 ms -> 16.22 ms -83% diagram_file re-reads the report rather than being handed the page's copy, which is sound because the renderer only ever looks at the unfiltered function_matches, library_matches and reference_sample_entry - and that is asserted rather than assumed: every diagram the pages display is byte-identical before and after, and a test renders the page-filtered and unfiltered reports and compares bytes, so a future mcrit that makes the renderer filter-sensitive fails here instead of silently drawing the wrong picture. The pagination count aggregated the whole report and threw it away, only for the template to build it again for the page slice. getAggregatedFunctionMatches groups by function_id and drops no group, so the count is the number of distinct function_ids - asserted across three reports and six narrowings, and those assertions pass on master too, because they prove the claim rather than the change. matches_for_sample 1913 matches, 756 distinct 2.433 ms -> 0.033 ms matches_for_query 2072 matches, 537 distinct 2.372 ms -> 0.034 ms load_cached_result matched `job_id in filename` against a name that also carries a timestamp, so load_cached_result(app, "2026") returned some arbitrary other job's report - and, having no reason to stop, it json.load()ed every match and kept the last. Matching the id against the part of the name it actually occupies and reading only the newest hit fixes both: the same job cached 20 times: 97.34 ms -> 1.45 ms I did not use glob() for this. I measured it, and at 20000 cached files it was slower than the scan it would have replaced. And the jobs pages re-fetched the same family once per job carrying it, with no `not in` guard - unlike the sample loop two lines above, which has one. 25 jobs in one family: 25 getFamily calls -> 1. Two pre-existing hazards fixed on the way, because this change makes both more likely to be hit: image.save() and the result cache both wrote in place while other requests were reading them, so a reader could be served - and browser-cache - a truncated PNG. Both now render to a temporary file and os.replace it into position. And a job id from the URL is validated before it is pasted into a cache path.
pytest and ruff used to arrive as transitive dependencies of mcrit and no longer do, so a clean checkout could not run `make test` or `make lint`. Install them explicitly in both the workflow and the target that claims to set the environment up.
Issue #80. The reported symptom - a YARA rule truncated to ~120 characters when copied - does not reproduce: a 5613-character rule copies intact. Two other defects in the same eight lines do. copyElementToClipboard built a detached textarea from $(element).html(). That is innerHTML, i.e. the markup the server rendered, so a rule the user had edited in the textarea was silently discarded and copied in its original form - and Jinja's escaping of & < > came back as & < >, which does not compile as YARA. The element passed in is already the textarea, so the fix is to read .value from it. Renamed to copyTextAreaToClipboard, since the old name is what invited .html(). While there: it prefers navigator.clipboard.writeText but keeps the select+execCommand fallback, because navigator.clipboard is not exposed on the plain-HTTP deployments this project ships for; it restores the caret; and it reports a failure instead of always claiming success. The statistics table gains a Version column. The report carries none - by_sample_id is block counts and nothing else - but a family job needs no extra request to find it: getFamily already answers with the family's samples and the caller is holding that entry. Only what the family did not supply is fetched by id, which for a job naming samples directly is one call per row. A sample the backend no longer has renders blank rather than failing the report. Its columns are sortable, client-side, over a table that is already fully in memory - so no round trip and no new query parameter to validate. Each numeric cell carries its raw integer in data-sort-value, so "2844 (66.76%)" still sorts as a number rather than as text. The sort symbols reuse the fa-sort ones cursor_pagination_sort_symbol already renders. DataTables was not used. Two sub-items of the issue needed no code. Sample coverage next to the picblockhash is already there - result_unique_blocks.html renders "{{ block["samples"]|length }} / {{ statistics["num_samples"] }}" and renderRule writes "coverage: 3/3 samples" into every block comment. Diversifying the block selection across function_ids belongs in mcrit: generateBlockCover sorts on (uncovered-sample count, block score) and never looks at function_id, and that is selection semantics. Also removed a dead guard: {% if family_id is not none %} around the Samples header, where family_id is never passed to render_template and Jinja evaluates "Undefined is not none" as True - so the column always rendered and the guard said nothing.
pytest and ruff used to arrive as transitive dependencies of mcrit and no longer do, so a clean checkout could not run `make test` or `make lint`. Install them explicitly in both the workflow and the target that claims to set the environment up.
pytest and ruff used to arrive as transitive dependencies of mcrit and no longer do, so a clean checkout could not run `make test` or `make lint`. Install them explicitly in both the workflow and the target that claims to set the environment up.
…gits The tooltip is the score to two decimals, so it is not the source value. A score of 2.501 renders as tooltip "2.50" and cell "3", and the assertion shown == round(tooltip) would have failed on a correct page - a spurious failure waiting for the next fixture regeneration, since these come from live reports. What the tooltip does pin is an interval: the score is within 0.005 of it. So the cell has to be within half a unit of that interval. Truncation is still caught, because it is off by the whole fractional part and these reports carry plenty above 0.505 - 85.88, 82.70, 42.81, 60.99. Mutation-checked: restoring %3d fails all three parametrisations. Found by Codex on the PR.
Issue #42 asks for three things. (a) a working "Reset order" turned out to be already implemented - dropping `?custom=` is the reset, and the view falls back to the backend's `clustered_sequence` - so it is only covered by tests here, not rebuilt. (b) and (c) are new: order the matrix by ascending sample ID, or by family name and version. The orderings are named (`?order=clustered|sample_id|family`) and resolved server-side rather than frozen into a `?custom=` id list. A cross compare over a couple of thousand samples would make that list several kilobytes of query string, and a named order stays correct after a family is renamed. `?custom=` still wins over `?order=`, so existing links keep working and the drag-and-drop handler needs no change: it reads the sample list, which renders in the active order, so a named order followed by a drag chains correctly. One order applies to all six matrices, as before; the page copy now says so instead of leaving it to be discovered. Two things the review turned up and this fixes. `family_sort_key` coerces the family and version fields with `str(... or "")` - both come out of an analysed binary, and a backend that stored a non-string there took the whole result page down with an `AttributeError`. And the invalid-`?custom=` path passed `job_info=result_json` to `result_corrupted.html`, which rendered "Results for Job: are corrupted" and a Delete button posting to `/data/jobs//delete`; every other corrupted render in the file passes `job_info`. `?order=` is whitelisted through `parse_choice_query_param`, matching how the other query parameters in this application are parsed. +37 tests: 17 page-level and 20 unit-level. Four of the page tests fail before this change.
Three corrections a review turned up.
The `{% if %}` around the label cell emitted a blank line either side and
re-indented the `<td>` on *every* page, including unfiltered ones - so the claim
that an unfiltered page is unchanged down to the rendered string was false. 27
rendered pages against master: 27 differed before, 10 after, and those 10 are the
pages this change is actually about (famid, samid, funid, exclude_library). All
three unfiltered pages are now byte-identical to master again. Fixed with
whitespace control.
The comment on `is_filtered` said "a score or library filter narrows the page
just as much". Only half true, and the half that is false is the interesting
one. `applyFilterValues` splits into family/sample filters and function filters,
and only the second group touches `filtered_function_matches`. Measured on the
captured 1-vs-N report:
filter_direct_min_score=20 3/11 samples 1913/1913 functions
filter_unique_only 7/11 1913/1913
filter_family_name=win.citadel 2/11 1913/1913
filter_exclude_library 6/11 1812/1913
filter_exclude_pic 11/11 576/1913
filter_func_unique 11/11 557/1913
The six that narrow only the sample list leave every function match in place, so
the statistics and the function table below them stay in step - both showing the
whole job's function matches. They do disagree with the *sample* table, which is
a real gap, but not one this table closes by relabelling: the numbers would still
be the ones it is showing. A parametrised test now pins all nine, because a
comment cannot go red.
The claim that no other integer rendering was on a score was wrong.
`MatchedFunctionEntry.matched_score` is a float, and `MatchingResult` maxes it
into the `best_score` the aggregated tables show, so six more cells truncated
exactly as the sample columns did - one of them the "Best Score" column, which
is in the default `UserColumnSettings` active list and therefore renders on the
bare result URL with no query parameters at all:
matches_for_sample result_compare_all.html 100 cells, 11 truncated 93.75->"93"
matches_for_query result_compare_all.html 100 cells, 15 truncated 76.5625->"76"
matches_for_sample_vs result_compare_vs.html 100 cells, 14 truncated 90.625->"90"
Across the fixture that is 203/1913, 235/2072 and 56/468 function-match scores
losing half a point or more - issue #7's own complaint, one table lower on the
same page.
The existing test could not see them: its locator requires a `Percent: NN.NN%`
tooltip and these cells have none. The new test finds the column by its header
with an HTMLParser and reads the exact value off the render context through
Flask's `template_rendered` signal, so it asserts `int(cell) == round(score)`
rather than a tolerance.
And the existing test rendered only two of the four templates this change
touches: `?famid=` and `?samid=` were never set, so `result_compare_family.html`
and `result_compare_sample.html` - 8 of the 20 already-changed cells - were
untested. The parametrisation is now (report, query, template) triples and both
tests assert which template the request actually reached, which took two
corrections against the fixtures: `?famid=` has to go on the 1-vs-N report
(the 1-vs-1 family page has an empty sample table), and `?samid=`/`?funid=` have
to go on the 1-vs-1 report (the by-id pool is only complete there).
Mutating each of the five table cells back to `%d` individually now fails
between one and two tests each; so does each of the four already-shipped cells
on the newly reached templates.
242 -> 250.
Four findings from a review of this change, plus the smells it turned up.
**The cached files silently became mode 0600.** `tempfile.mkstemp` creates at
0600 by design - it is built for secrets - and `os.replace` preserves that,
where master's `open(path, "w")` produced the umask's answer. Measured on a real
render: the report JSON and both diagrams came out 0o600 against a 0o644
reference. Any deployment where nginx serves `instance/cache` directly, or where
a different uid reads it, breaks.
Fixed by not using `mkstemp`: the file is opened through `open(..., opener=...)`
asking for mode 0666, which `open(2)` masks with the umask in the kernel - so it
reproduces the in-place write exactly, with no `os.umask(0)` window to race and
no chmod. Measured: umask 022 -> 0644, 027 -> 0640, 002 -> 0664. A hardcoded
0644 was rejected because it would *widen* a hardened deployment. The opener also
adds `O_EXCL`, and using `open` rather than `os.open` + `os.fdopen` closes an fd
leak the previous version had on the `fdopen` failure path.
**`write_atomically` was completely untested.** Reverting both call sites to
master's in-place writes left the whole suite green, and the two tests named for
atomicity could not see it: one monkeypatched `renderStackedDiagram` to raise -
i.e. *before any write happens* - and the other asserted a file exists and
parses, which is true in place too. Both are replaced by tests that fail *during*
the write: a `HalfWrittenImage` that emits the PNG magic, flushes, then raises,
and a payload whose non-serialisable member makes `json.dump` emit ~50 bytes and
then raise. Against the in-place revert the diagram one now fails with
`assert 200 == 404` - the truncated PNG was actually served, which is the hazard.
Neither old body was deleted; both were kept under names that say what they
really test.
**The temp files sat in the directory the app serves.** `mkstemp(dir=cache_path)`
put them under `cache/diagrams`, which `diagram_file` serves for any `<path:filename>` -
a leftover `.incomplete-*.part` came back `200` with its partial content. They now
go to `instance/cache/incomplete/`, a sibling of the two cache directories. Chosen
over rejecting the `.part` suffix in the route because the hazard is "an unfinished
file is reachable", not "its extension is .part". Six traversal spellings checked
against the route, all 404.
**A non-canonical filter id did unbounded uncacheable work.** `DIAGRAM_FILENAME_RE`
accepted `-?\d{1,18}`, but the file written is named from `int(filter_id)`, so
`-famid_0001` rendered `famid_1` and then served 404 for the name asked for -
repeating the report parse on every hit, forever, behind nothing but
`@visitor_required`. The grammar is now `0|-?[1-9][0-9]{0,17}`, exactly how
`str(int)` spells it, which makes filename -> (job, filter) injective so the
on-disk short-circuit can always answer. 78.8ms and a `MatchingResult.fromDict`
per hit becomes 1.5ms and none. All four templates emit `'%d' % famid`, so
nothing the app produces changes.
**`load_cached_result(app, job_id)` logged through `current_app`** on its
unreadable-file path, despite taking `app` explicitly so it would not need a
request context. `RuntimeError: Working outside of application context.`
Also: two of the four parametrisations in
`test_the_diagram_does_not_depend_on_the_filtering_the_page_applies` were
vacuous - `{}` and `filtered_function_id: 0` narrow nothing, so both sides were
trivially equal. A second axis over user filters takes the mutation from 2/4
caught to 5/8. And the stale docstring claiming a job id is "pasted into a glob
pattern" - there is no glob anywhere in `mcritweb/` - now says the metacharacters
are there as a shape a reader expects refused.
326 -> 345.
Both things this change exists to do could be reverted with the suite green. Restoring `copyTextAreaToClipboard` to the `$(element).html()`-into-a-detached- textarea implementation that issue #80 is *about*: 245 passed. Deleting the entire client-side sorting script: 245 passed. The sort test asserted three substrings - `data-sort="number"`, `data-sort="text"`, `data-sort-value="2844"` - which is markup, and its docstring claimed "sorting happens in the page" while nothing verified that. Both now have a lint that always runs and a browser test that drives the behaviour, because either alone is insufficient: a lint never executes the script, and a browser test leaves CI with no cover at all, since playwright is not a dependency here. The lint checks that the icon's onclick names the helper *and* that the id it names exists (a helper wired to the wrong selector is as broken as none), that the helper reads `textarea.value`, and that its code carries none of `copyElementToClipboard`, `.html()` or `.innerHTML` - comments stripped first, because the helper's own comment names the old implementation. `tests/testBrowser.py` boots the same offline harness the suite already uses on a loopback port and drives Chromium. The clipboard test *edits* the rule to `rule edited { condition: 1 < 2 and 3 > 2 /* R&D */ }` before clicking, so the assertion is about the current value rather than the rendered one; under the old implementation the clipboard carries the shipped rule instead of the edit. The sort tests read `textContent`, never `dataset.sortValue`, so they are not asserting the implementation back at itself. It skips cleanly where playwright or Chromium is absent - both paths verified, exit 0 either way. And the "a family job needs no extra request" claim was dead in the suite. `tests/fixtures/families.json` was captured off `/families`, which cannot carry samples - `MinHashIndex.getFamilies()` reads storage, and only `FamilyResource.on_get` fills them in - so `FamilyEntry.samples` was None in every integration test and the shortcut never ran: Counter({'getSampleById': 3, 'getJobData': 1, 'getResultForJob': 1, 'getFamily': 1}) `regenerate.py` now captures `/families/{id}?with_samples=true` per family, the fixture is rebuilt to match, and the fake honours `with_samples` so the two endpoints stay distinguishable. The claim is asserted on the rendered page: `getFamily == 1`, `getSampleById == 0`, and a version still present so "no requests" cannot mean "no versions". That fixture change made `test_unique_blocks_page_survives_a_backend_that_lost_a_sample` vacuous - it stubs `getSampleById -> None`, which is no longer called on the family path. Rewritten rather than deleted: it now drops a sample from the family so the by-id lookup is reached, and asserts that sample's version is absent while the others are present. Six mutations, none surviving: the old clipboard implementation (2 failed), the sorting script deleted (3), the helper renamed but still copying innerHTML (2), the family shortcut removed (3), the fixture recaptured off `/families` (3), and the fake serving one shape from both endpoints (1). 245 -> 251.
write_atomically opens the report cache in text mode, and that cache is served back verbatim as the raw result - so what lands on disk has to be byte-for-byte what json.dump wrote. Text mode does not promise that: it rewrites the newline on any platform whose line ending is not LF, which on Windows makes every one a CRLF and the served file stop matching the bytes it claims to be. newline="" for the text modes; it is not a valid argument in binary mode, hence the conditional. Found by merging this branch with the raw-result download (#75), which compares the served bytes against the backend's and fails on the difference. Neither branch can see it alone - this one writes the cache but nothing here checks its bytes, and that one checks the bytes but writes in place. The new test is honest about its own limit: on Linux, where text mode translates nothing, it passes with or without the fix, so CI cannot see the regression it guards. Mutation-checked where it does bite - dropping newline="" fails it.
The statistics table already followed `filtered_function_matches`, which covers `?famid=`, `?samid=`, `?funid=` and the three function-level filters. It did not cover the six filters above the sample table: `applyFilterValues()` keeps two lists and those six rebind only `filtered_sample_matches`, so the table stayed job-wide. That left the same view reachable two ways with two different answers. On the captured 1-vs-corpus report, `?famid=3` and `filter_family_name=win.dridex` show the identical pair of win.dridex samples; the first said 4 own functions / 249.0 bytes and the second said the whole job's 756 / 151654.0. So define "on screen" as the intersection: a function match counts when its own function survived the function-level filters *and* the sample it matched is still in the sample table. That is the definition `filterToFamilyId`/`filterToSampleId` already imply, since they narrow both lists in lockstep, and it costs nothing unfiltered - `_aggregateMatchSampleSummary` derives the report's sample list from the same function matches, so no `matched_sample_id` is ever missing from it. The equivalence with the backend's own aggregation is unchanged and still asserted for every captured report. The trade is stated in the helper: for those six filters the statistics now describe less than the function table below them, which mcrit still renders job-wide. The page's subject is the samples it is willing to show, so that is the honest direction. `test_which_filters_the_statistics_follow` is rewritten rather than dropped. It recorded which list each filter narrows and asserted the statistics ignored six of them, which is the behaviour this commit removes; it now records the same split and asserts the statistics follow either way. Its indicator field moves from `num_own_functions_matched` to `num_foreign_functions_matched`, because a foreign function belongs to exactly one matched sample while an own function usually matches in several, so the own-function count can sit still through a real narrowing. Also pins `?samid=` at page level, which the PR had recorded as untestable offline. It is untestable on the 1-vs-corpus and query reports, where `assign_matched_offsets` reaches past the captured pool - but a 1-vs-1 report matched one sample, so the pool that is complete for the report is complete for its `?samid=` page too. tests/fixtures/README.md now says which of the filtered views render and which do not.
Two of the YARA half of issue #80 were still open. The Block column of the unique blocks table is YARA syntax and is there to be copied out, but it was wrapped by breaking every 80th character. That lands mid-byte far more often than not: of the 51 sequences in the captured report long enough to wrap, 41 were cut inside a token and 23 of those left an odd-length run - "6a33" arriving as "6a3" and "3" - which no YARA compiler will take. Short sequences never wrapped, which is why the issue saw it only on "extensively long yara strings". It now wraps between bytes with mcrit's own wrap_string, the way renderRule already wraps the rule itself. The rule names the picblockhash and its sample coverage but not the function the block sits in. mcrit builds the cover greedily by how many uncovered samples a block adds, tie-broken on score, with no notion of function identity, so nothing stops a rule from fingerprinting a single function - and `condition: 7 of them` fails the moment that function is recompiled. The captured report spreads its ten blocks over seven functions, so it is not degenerate here, but a reader cannot tell. Of the two things the issue offers - include function_id, or make the cover prefer a variety of them - this takes the first: which blocks to pick is mcrit's decision to change, and forcing variety here would mean reimplementing its set cover and trading sample coverage away for resilience this data cannot calibrate. Naming the function costs no coverage and makes the spread readable before the rule ships. Getting at the cover means calling generateBlockCover and renderRule instead of generateYaraRule, which is those two lines and then discards the cover. The comment is rebuilt exactly as renderRule writes it and replaced by name, so an upstream format change leaves the rule as it was rather than mangling it.
Rounding the columns fixed a contributor to "too far from the expected value" but not the verification the issue asks for. The arithmetic is in mcrit, but the captured reports carry both its inputs (matches.functions: every function with its byte weight, instruction count and match list) and its outputs (matches.samples[*].matched.percent), so it is checkable from here. An independent transcription of _aggregateMatchSampleSummary reproduces all 150 shipped percentage cells across the three fixtures exactly. The calculation is correct. What is wrong is around it, and the largest piece is ours: all twenty score tooltips render "Bytes: X / binweight" above "Percent: Z%" where Z is not X/binweight - the percent divides by matchable bytes, or for the nonlib columns by matchable minus library. On the top match that offers the reader 84.28 and states 85.79. The gap scales with the sample's library share, so it is unbounded, where truncation was off by at most 0.994 across those same 150 cells. Truncation therefore cannot have been the whole answer. Also found, in mcrit, reported not fixed: MatchedFunctionEntry stores flag bits under boolean names, so getMatchTuple maps flags 2 -> 4 and 7 -> 21 and a toDict/fromDict round trip turns a pichash match into a library one. Latent - MatchingResult.toDict has no production caller in either repo - but it does explain our own standing TODO at MatchReportRenderer.py:249, since a pichash match scores 100 and match_is_pichash contributes 2. The test is a characterisation test, not a regression guard: it passes before and after, and what it catches is a re-captured fixture that no longer agrees with the formula the ADR documents. Five mutations of the transcription - wrong denominator, no family adjustment, > for >= at the size threshold, nonlib sharing the plain denominator, and dropping the score factor - each fail it. 250 -> 256.
Two of the three boxes issue #51 opens with, the search box having been done already. "sort should be done in mcrit and not only consider displayed elements": the page's only sorting apparatus was a DataTable configured against '#job-table' while the table is rendered with id="{{ active }}", so it never initialised, and had it initialised it would have reordered the 25 rows on screen and left the rest of the queue alone - the complaint verbatim. It is gone, replaced by a link on the '#' column that flips `ascending`, which mcrit does apply to the whole queue: mongoqueue's get_jobs sorts by _id before it skips and limits. That is also all mcrit can do - nothing from getQueueData down to the collection accepts a sort key - so Type, Started, Finished and Progress stay honestly inert rather than offering an order that stops at the page boundary. Unlike the search, this costs nothing: the browse path still fetches one page. "what about cross jobs?" had a concrete answer rather than a design question. A cross compare runs as one getMatchesForSampleVsGroup job per sample plus a combineMatchesToCross job that merges them, and mcrit's Job.method_types - which is where this page's category menu comes from - lists the group jobs neither under "matching" nor in "all". So the jobs a cross compare does its work in had no tab: they could not be browsed, ordered or searched, and the "Matching" count left them out while the totals row above it counted them. Both now include them. The order link carries only the five parameters the page understands. Built from request.args wholesale it would hand url_for its own reserved keys straight from the query string, and it drops the page number, since page 3 of one order is an unrelated slice of the other. New tests/testJobSort.py, eight cases, all eight failing before this change.
ADR-0003 established that the arithmetic behind the nonlib frequency score is correct. #7 reports a value too far from the *expected* value, so the expectation is what was wrong - and the hover text is what sets it. All twenty score tooltips printed Bytes: 130682.88 / 155065 <- reference_sample_entry.binweight Percent: 85.79% where the percentage is taken against a different total: matchable bytes (153466) for the plain columns, matchable minus library-matching functions (152337) for the nonlib_ ones. Neither is the binweight, so the reader was offered 84.28 and told 85.79. The gap scales as binweight/(matchable-library): 1.5 points here at a 0.7% library share, and unbounded as that share grows. Neither total is on the wire and neither can be recomputed here - the function list carries no instruction counts and omits unmatched functions. But the divisor is a property of the reference sample, identical on every row, so it does not need recomputing, only inverting: 100 * matched_bytes_X / matched_percent_X recovers it exactly from any row that scored above zero. recover_score_divisors() does that once per render, off the unfiltered matches so a ?famid=/?samid= page agrees with the unfiltered one, and the page duplicates none of mcrit's threshold logic. A group where every row scored zero cannot be inverted; it falls back, and a zero numerator reads as 0% over any total, so no cell is left inconsistent with its neighbours. Mutation-checked per cell, not just in bulk. Reverting all twenty fails all five pages; reverting one column across the four templates fails all five; reverting one template fails exactly the pages that reach it - all 8 cells of result_compare_all.html fails 2, and family, sample and vs fail 1 each. Also does the browser walk AGENTS.md asks for and the branch had skipped. tests/testBrowser.py drives Chromium over the four templates, because the offline tests read HTML as text and so cannot see the two things that only exist once a browser has the page: has to become a newline in the attribute, and hint.css draws the tooltip from content: attr(data-hint), which is in no markup. Both checked, plus no console errors. It skips where playwright is absent, which includes CI - a browser download on every job of a four-version matrix is not worth one hover text, and AGENTS.md now says so and says to run it locally. The two mcrit defects in ADR-0003 stay reported, not fixed. 256 -> 263.
fix/72 holds 0003 and several branches in this round each wrote one. Different filenames, so git merges them all without conflicting and the tree ends up with ADRs sharing a number.
Resolutions: - __init__.py: master's client imports and #142's matching_statistics; both the search-highlighting filters (#136) and #142's template global. - result_compare_*.html: #244's heading levels with #142's recomputed statistics. - tests/fixtures/README.md: #142's account of which filtered views the pools cover, then master's cfg_loop_functions.json section.
Every id in ?custom= becomes a row and a column of each matrix, and the lazy panes (#213) carry the same rows as JSON, so one id repeated 200 times already rendered a 12.5 MB page, and an 8 KB request line would ask for millions of cells. order_samples (#145) now refuses a repeated id, which shows the existing "cannot produce the chosen custom ordering" page. #159's page text no longer claims the clustered order is shared by all matrices: mcrit clusters each matching method separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
Resolutions: - views/data.py jobs and job_by_id, __init__.py index, views/explore.py: master's batched lookups (#191) and #145's family guard, each over #115's describable_jobs, so a job whose payload cannot be read loses its own lookups, not the page. The index and the job overview now apply that guard to the batched calls as well. - jobs.html: #115's no-match message for a search; otherwise the table with #115's order toggle and #119's per-category empty states. - __init__.py / utility.py: imports and template globals from both sides.
- the job list tests follow #132's redirect to the named tab, which keeps Search, ascending and p in the URL; - the malformed-payload test backend answers the batched getSamplesByIds and getFamiliesByIds with nothing, so the views fall back to the single calls; - the corpus fake selects sample_ids on payload.descriptor as mcrit's mongoqueue does, instead of parsing every job, so an unreadable job still reaches the listing the test is about rather than failing inside the fake. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
… matched #145 moved the diagram to its own GET, which checked only that a family or sample exists. Any visitor could then render one PNG per family and per sample in the corpus for every job, into a diagrams cache nothing bounds. The route now renders a famid/samid diagram only for a family or sample the report's sample matches name, which covers every filter the result page links to (library samples included), and it no longer needs the isFamilyId/isSampleId round trips. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
Resolutions: - views/analyze.py: master already moved each sample search to the top of its view; #174's copies at the old places are dropped and master's calls go through search_page. The hash-list redirect never rendered its search, so it gets none. - __init__.py index and views/explore.py: master's batched lookups, guarded by #115's describable_jobs, with #174's typed entries for the searches. - views/explore.py keeps the FamilyEntry import #174 dropped: #219's rename read-back (family_read_back) builds one.
Resolutions: master's mcrit>=1.12.0 pin and Python 3.12 floor stand (#212 raised them to 1.9.0 and 3.11, which master has since passed); AGENTS.md names #212's testMcritFloor.py beside master's pin; the changelog entry waits for the release; data.py JOB_CATEGORIES carries #115's getMatchesForSampleVsGroup and #212's three repair jobs.
#115 filters the category it fetched, and turned a failed read into an empty list, so the page reported "0 jobs matching" and "No job's parameters contain ..." for a backend that never answered. It now flashes the failure and shows neither sentence. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
…nd name their jobs once - every scheduling button on the server page asks for confirmation and disables itself on submit: each forces a recalculation over the whole corpus, so a stray or double click queued it, or queued it twice; - the six scheduling routes carry mcrit_server_required: McritClient lets a connection error propagate, so an unreachable backend was a 500, not the message the helper's docstring promised; - the three repairs' empty states say an admin can start them from the server page, which #212 made true; - the job list names them through the job_name filter like every other job, so it and the job pages agree ("Repair stale MinHashes", as on the button). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
This was referenced Oct 19, 2026
r0ny123
added a commit
to r0ny123/mcritweb
that referenced
this pull request
Oct 20, 2026
The entries for familiary#298 and familiary#299, from their change lists and their notes for deployment, and familiary#212's entry, which was held back for this. familiary#212's floor change is not repeated: master's mcrit>=1.12.0 was already in 1.6.0. Fixes made during the two integrations to code that had not been released are left out.
r0ny123
added a commit
that referenced
this pull request
Oct 20, 2026
Conflicts in 23 templates and stylesheets, where batch C (#298, #299) changed the same lines as this branch. Master's changes are kept and this branch's are applied on top: - Headings: #244's h2/h3 outline wins over this branch's restyled headings; the delete-job modal title is <h2 class="modal-title h5">. - Job titles: master's job_name filter replaces job_info.parameters in the link hunt, maintenance and function result headings, keeping the breadcrumb this branch added. - Empty states: master's empty_state macro replaces this branch's inline messages; the macro itself now uses a centred div instead of <center>, and so does the job search's no-match line. - Cross compare: master renders one method's matrix and fills the other tabs on click. Its tab loop now shows this branch's labels, looked up by method name; the pane body is a div instead of <center>, and the script finds it by class. - Admin page: the maintenance forms keep master's confirm-before-submit handler and this branch's spacing classes. - Job errors: master's pre.job-error replaces the wrapper div. - Row buttons: this branch's per-item aria-labels stay, with master's titles added where this branch had none. - YARA copy: the button stays and calls master's copyTextAreaToClipboard; testResultPages accepts a button carrying that handler, not only an <i>.
This was referenced Oct 20, 2026
This was referenced Oct 24, 2026
Merged
r0ny123
pushed a commit
to r0ny123/mcritweb
that referenced
this pull request
Oct 30, 2026
Bumps setup.py and adds the version history entry, folding in the five "unreleased" lines (familiary#263, familiary#262, familiary#261, familiary#293, familiary#254). 48 PRs since 1.6.0: batches C1 (familiary#298), C2 (familiary#299), security E (familiary#318), UI F (familiary#339) and fixes G (familiary#340). No dependency changes. The entry leads with the three things that affect existing deployments: instance/ backups under WAL mode, the bounded result cache, and the optional retirement of pre-1.5.0 API tokens. Test count measured on the batch G tree, which master 30cbc8a is identical to: 2428 passed, 19 skipped against mcrit 1.16.0, 2447 collected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch C2 (UI), stacked on C1 (#298): thirteen batch C PRs that conflicted with master after batches A, B and D, merged with master and with each other. They are integrated the same way as #295: each PR's head commit is an ancestor of this branch, so GitHub marks them merged when this lands with a merge commit. This PR targets
integrate/ui-C1, so its diff shows C2 alone. Merge the C1 PR first; this one then retargets tomaster. Closes the "merge master in" ask of #249 for the PRs listed here.RESULT_CACHE_MAX_BYTES/_FILES) and written as compact JSON (after #145)views/search.py)Resolutions
Most conflicts are with master's batch lookups (#191,
get_sample_entries/get_family_entries/dependency_jobs) and with #194's paged aggregation.describable_jobsguard now wraps the batched calls on the index, the jobs page, the job overview and the sample listing.views/data.py): Render the match diagram off the page request, and stop doing the work twice (#68) #145'scount_aggregated_function_matchesanddiagram_sizesit alongside master'sfunction_rows(Result views rebuild function aggregations and sample-match passes repeatedly #194) and Round the score columns instead of truncating them (part of #7) #148'sdivisors. The sample branch drops the in-requestcreate_match_diagramand Render the match diagram off the page request, and stop doing the work twice (#68) #145's secondgetSampleById, because master's branch condition already looks the sample up. Render the match diagram off the page request, and stop doing the work twice (#68) #145's one-passorder_samplesserves Offer the cross compare matrix in three named orders (#42) #159's orderings.job_stateis passed to master's overview.job_column_tabletakesprogress_id.cross-lazy-panesJSON. Render only the visible cross compare matrix, fill the other tabs on first click #213's AGENTS.md note links ADR-0014 (its side still said ADR-0003).write_atomicallywrites Bound the result cache and write cached reports as compact JSON #229's compact JSON, then the trim runs. Bound the result cache and write cached reports as compact JSON #229's AGENTS.md side still carried the pre-Make the per-role upload size cap configurable #19 "uploads named by SHA-256" bullet, which is dropped.tests/testBrowser.py. Round the score columns instead of truncating them (part of #7) #148's becomestests/testBrowserScoreTooltips.py, and AGENTS.md names both.mcrit>=1.12.0and Python 3.12 floor stand (Add admin controls for the three corpus repair jobs #212 had raised them to 1.9.0 and 3.11). Its changelog entry waits for the release.match_diagrammacro and the recomputed statistics together.Fixes made during integration
get_sample_entriesbatch, not onegetSampleByIdper row. Found in the mcritbox integration (ref/ui-C-mcritbox-20260930).?custom=: an order repeating a sample id is refused. Every repetition becomes a row and a column, and the lazy panes carry the rows again as JSON: one id 200 times rendered a 12.5 MB page. Offer the cross compare matrix in three named orders (#42) #159's text no longer says the clustered order is shared by all matrices, since mcrit clusters each method separately.famid/samiddiagram only for a family or sample the report matched. Checking that one merely exists let a visitor render one PNG per family and per sample in the corpus, for every job, into the unbounded diagrams cache. This also saves theisFamilyId/isSampleIdround trips.mcrit_server_required, because an unreachable backend was a 500;job_name.Search,ascendingandp;sample_idsonpayload.descriptor, the way mcrit's mongoqueue does;Verified
ruff check(0.16.0) clean.textContent/setAttributeonly, and family names are escaped everywhere they appear. They also checked that the#132redirect cannot be made into an open redirect, that Consume searches as entry objects through one adapter #174 behaves as before for every former consumer, autocomplete escaping included, and that Narrow the matching statistics to the matches on screen (#38) #142's statistics match mcrit's aggregation when unfiltered.For deployment
Known, not fixed here
filterafter paging. mcrit 1.16 filters before paging and hasgetQueueCount, so the search can be paged on the backend for 1.16+.Not in this batch
band_df_cutoff(andshortlist_sizefor 1-vs-N), which the rerun allowlist rejects, so no job gets a rerun or configuration link. Review on Offer a rerun on the jobs a request can be faithfully rebuilt from (#55) #133.Nonefor a failed read (dependency_jobs, the batch-lookup fallback, Confirm sample/family modifications instead of sleeping a fixed time #219's read-back). Needs a decision per call site. Review on Answer a failed backend call with a page, not a stack trace (#43) #130.🤖 Generated with Claude Code
https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG