Skip to content

Integrate batch C2 (UI): #132 #147 #148 #214 #159 #213 #226 #145 #229 #142 #115 #174 #212 (stacked on #298) - #299

Merged
danielplohmann merged 80 commits into
masterfrom
integrate/ui-C2
Oct 19, 2026
Merged

danielplohmann merged 80 commits into
masterfrom
integrate/ui-C2

Conversation

@danielplohmann

Copy link
Copy Markdown
Collaborator

Batch C2 (UI), stacked on C1 (#298): thirteen batch C PRs that conflicted with master after batches A, B and D, merged with master and with each other. They are integrated the same way as #295: each PR's head commit is an ancestor of this branch, so GitHub marks them merged when this lands with a merge commit. This PR targets integrate/ui-C1, so its diff shows C2 alone. Merge the C1 PR first; this one then retargets to master. Closes the "merge master in" ask of #249 for the PRs listed here.

PR change
#132 the job list names its tab in the URL, so refresh, back and links reproduce the page
#147 the YARA copy button copies what the textarea holds; the statistics table has a Version column; rule comments name each block's function
#148 score columns round instead of truncating; tooltips state the fraction the percentage is
#214 a running job's overview polls its status and reloads only when something changed
#159 the cross compare matrix in three named orders (clustered, sample id, family)
#213 only the visible cross compare matrix is rendered; the other tabs fill on first click
#226 cross compare tooltips are built on hover, and a job's samples are grouped in one pass (on #213)
#145 the match diagram renders on its own request into a reserved box; aggregation counted once; whole-name cache matching; atomic cache writes
#229 the result cache is bounded (RESULT_CACHE_MAX_BYTES / _FILES) and written as compact JSON (after #145)
#142 Matching Method Statistics are recomputed for the matches on screen
#115 the jobs page search box works; ordering by job number over the whole queue; a tab for the per-sample jobs a cross compare runs; one unreadable job no longer takes a listing down
#174 searches are consumed as typed entries through one adapter (views/search.py)
#212 the server page schedules the three corpus repairs mcrit 1.9.0 added

Resolutions

Most conflicts are with master's batch lookups (#191, get_sample_entries / get_family_entries / dependency_jobs) and with #194's paged aggregation.

Fixes made during integration

Verified

For deployment

Known, not fixed here

Not in this batch

🤖 Generated with Claude Code

https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG

r0ny123 added 30 commits August 30, 2026 02:07
The search half of the issue was half-built and inert: the form in jobs.html sat
inside a Jinja comment, so there was no search box at all, and data.jobs still
read request.form['Search'] into a `query` it passed to the template and never
used for anything.

McritClient.getQueueData takes a `filter` string, which the backend applies as a
substring test against each job's parameters, so the term now has somewhere to
go. The box is back, as a GET rather than a POST: a search is a read, and in the
URL it is linkable, survives a refresh, and is carried by the pagination links,
which build from request.args. The route is GET-only again as a result - the POST
branch also raised BadRequestKeyError for any POST without the field.

The category, state and sort order ride along as hidden fields, so searching
narrows the list you were looking at instead of resetting it.

The sort half of the issue was already done, in an earlier release.

One test case disappears with this: testCsrf's form scanner was matching the
commented-out POST form. There is no POST form in jobs.html any more.
Every unit job on every Python version fails with:

    /opt/hostedtoolcache/Python/3.11.16/x64/bin/python: No module named pytest

The Makefile said "pytest and pytest-cov arrive with mcrit, so requirements.txt
covers both". mcrit declares them under its `dev` extra:

    $ python -c "import importlib.metadata as m; print(m.requires('mcrit'))"
    ... 'pytest; extra == "dev"', 'pytest-cov; extra == "dev"' ...

so a plain `pip install -r requirements.txt` does not bring them in, and the
suite has not actually run in CI since mcrit moved them there. Ruff is green
throughout, which is why this looked like a working pipeline.

Installed in the workflow rather than added to requirements.txt: neither is a
runtime dependency and a deployment should not get them. Pinned, for the same
reason the lint job pins ruff. The two stale comments that said otherwise are
corrected.

(cherry picked from commit 8268666)
work/harness/cookies.txt is a cookie jar written by a local debugging
harness - a Flask session signed with a secret hardcoded in that harness,
against a throwaway SQLite database. Not a credential for anything, but it
has no business in this change. It arrived with the cherry-picked CI fix.
…at once

Review caught that passing the search term to getQueueData's own `filter` is
not just imprecise, it makes the feature worse than absent. mcrit slices the
requested page and filters what survives:

    return [job._data for job in self.queue.get_jobs(start_index, limit,
            method, state, ascending) if filter in job.parameters]

so `start=0, limit=25, filter=x` answers "the matches among jobs 0-24", not
"the first 25 matches". With 100 jobs and the only hit at number 60, page 1
of the search rendered empty and offered four pages counted from the
*unfiltered* total - the match was reachable only by guessing which page it
was on, and page 1 read as "no such job". I noticed the ordering, wrote a
comment about it, and shipped it anyway. That was wrong.

A search now fetches the category unpaged and filters here, so the count
driving pagination is the count of rows actually shown. Browsing without a
search term still uses the paged call and is unchanged.

Two things fall out of that. The term no longer reaches the backend at all,
which also removes it from a URL McritClient builds by f-string with no
encoding - a `#` in a term used to truncate the query and silently drop the
state filter, and an `&state=failed` used to inject a second state parameter.
And the filter is now case-insensitive, because doing it here means choosing,
and a search box that misses EVIL.EXE for evil.exe is a surprise.

The empty state is search-aware: "No jobs available. Click here to create
your first job" is the wrong thing to say to someone whose search missed on
a full queue, and it reads as "there are no jobs", which is false.

The cost is that a search fetches the whole category rather than a page.
That is the price of a correct answer; the alternative is a filter in the
backend that runs before the slice, which is an mcrit change.

tests/testJobSearch.py now carries PagedQueue, a fake with mcrit's real
slice-then-filter semantics. The corpus fake ignores start, limit and filter
and returns the whole queue, which is why nothing caught this. 7 of its 14
tests fail against the previous implementation.
Re-review of the previous fix: `(job.parameters or "")` guards against None,
but Job.parameters never returns None - it returns "" or raises. It calls
json.loads on payload["params"] and then .items() on the result, so a
document whose params is malformed, `null`, or a JSON array raises
JSONDecodeError, AttributeError or TypeError.

That matters because filtering the whole category means the search now
touches every job in it. With one corrupt job at index 80 of a hundred:

    before this branch:  ?Search=x         200      ?p=4  500  (browse)
    with the filter:     ?Search=x         500      every page of the search

So the fix turned a one-page browse failure into a whole-category search
failure. The `or ""` shows I had considered None and not the raise.

A job whose parameters cannot be read cannot contain the search term either,
so leaving it out is not a workaround - it is the right answer. The read is
logged so the corrupt document is still visible to an operator.

The pre-existing browse failure is untouched: that one dies in the template,
which reads the same property, and fixing it is a different change.
`init` still ran `pip install -r requirements.txt` alone, so a fresh checkout
that follows the README lands without pytest and `make test` fails with
"No module named pytest" - the same failure the CI change on this branch fixes
for the runner, left unfixed for a contributor.

pytest and pytest-cov are not runtime dependencies and are not in
requirements.txt; mcrit declares them under its `dev` extra, which nothing
here requests, so they do not arrive transitively either. Install them from
`init`, pinned to the versions CI uses, and say so once above the target
rather than in a comment further down that describes a step that never ran.

Reported by Codex review.
The job list picks a tab by walking Job.method_types["all"] and taking the
first category the live queue statistics mention. That choice appears nowhere
in the URL - and the URL is the one thing a browser replays for a refresh or a
step back in history, so neither can reproduce the page. The tab also moves on
its own as soon as the queue changes underneath it.

When the tab was chosen implicitly, redirect to the URL that names it, keeping
the rest of the query. Every job-list history entry then carries its own tab,
and F5 lands where it was.

'active' is now validated before it is used to index the statistics, which it
was not: an invented category and a real-but-unqueued one both answered 500,
and the second is a link this very view renders. The accepted set is
method_types["all"], plus whatever the queue reports, plus the two maintenance
categories the menu below links to that mcrit's own method_types omits.
statistics.get() for the pagination count, so a category with no jobs renders
an empty tab instead of a KeyError.

The redirect builds its query with urlencode rather than passing the user's
args to url_for as keyword arguments: the names url_for reserves are user
input here, and ?endpoint=x is otherwise a TypeError. A loop is impossible -
the default is drawn from the static method_types["all"], so it validates on
the next request.

Considered and rejected: a history.replaceState shim in the template, which is
JS-dependent and leaves the server's own choice non-deterministic, and
remembering the tab in the session, which makes the URL less authoritative
still and breaks two job lists open at once.

Refs #36.
`init` still ran `pip install -r requirements.txt` alone, so a fresh checkout
that follows the README lands without pytest and `make test` fails with
"No module named pytest" - the same failure the CI change on this branch fixes
for the runner, left unfixed for a contributor.

pytest and pytest-cov are not runtime dependencies and are not in
requirements.txt; mcrit declares them under its `dev` extra, which nothing
here requests, so they do not arrive transitively either. Install them from
`init`, pinned to the versions CI uses, and say so once above the target
rather than in a comment further down that describes a step that never ran.

Reported by Codex review.
A match report carries one job-wide `match_aggregation`, computed by the
backend. `MatchingResult.fromDict` copies it through verbatim and no filter in
`applyFilterValues` touches it, so a result page narrowed to one family stated
the numbers of the whole job. On the win.dridex view of the captured report
that meant claiming 756 functions / 151654 bytes where the honest answer is
4 / 249. Issue #38.

Four of the five fields are counts over the very function matches the report
already contains, so aggregating the filtered list describes exactly what is on
screen and needs nothing from the backend. That is what makes this presentation
rather than a second opinion on the matching, which this repository does not
get to have - and the warrant is that the recomputation reproduces the
backend's own numbers exactly when nothing is filtered, for every captured
report, asserted in the suite so it fails loudly if it ever stops holding.

num_self_matches cannot be recomputed: _summarizeMatches drops matches of the
sample against itself from the function list before serializing it. It is
carried through job-wide and labelled "(whole job)" on a filtered page, rather
than passing a job-wide number off as a filtered one. Dropping the row instead
would delete real information on exactly the pages where someone is drilling
in, and make the table's row set depend on the query string.

is_filtered is the length of the filtered list against the full one, so a score
or library filter narrows the table too - those left it equally wrong before.

Two things worth recording because they were checked rather than assumed: the
report's per-match flags exactly reconstruct membership in the backend's
pichash/minhash sets, including under PICHASH_IMPLIES_MINHASH_MATCH, so the
split is faithful; and bytes_matched sums in a different order from the
backend's set, which is exact rather than merely close because all 2350
num_bytes values across the three reports are whole numbers far below 2**53.
pytest and ruff used to arrive as transitive dependencies of mcrit and no
longer do, so a clean checkout could not run `make test` or `make lint`.
Install them explicitly in both the workflow and the target that claims to
set the environment up.
The four score cells on every match result page rendered the percentage with
"%3d", which truncates toward zero rather than rounding: a sample scoring 85.88
showed as 85, a whole point below the 85.88 the tooltip on the same cell reports,
and 0.76 showed as 0 beside a neighbour at 1.04 that had barely outscored it.
That display gap is the shape of the "too far from the expected value" in #7 -
the scores themselves come off MatchedSampleEntry unchanged and are the backend's
to compute, not ours.

"%3.0f" keeps the column width and the right-alignment, matches the "%5.2f" the
tooltip beside it already uses, and rounds to nearest. Applied to all four score
columns (direct, direct_nonlib, frequency, frequency_nonlib) in all four result
templates so the columns stay comparable to each other.

Refs #7
…k twice

Issue #68. Four separate costs, measured on the offline corpus.

The diagram dominated the first view of every result page. create_match_diagram
ran synchronously at four points before render_template, and dragged in a
getFunctionsBySampleId round trip with it. The image already had its own route
and its own <img> request, so the work belongs there:

    /data/result/<sample>          102.05 ms -> 23.55 ms   -77%
    /data/result/<sample>?famid=1   93.53 ms -> 24.30 ms   -74%
    /data/result/<sample>?samid=1   84.95 ms -> 16.09 ms   -81%
    /data/result/<sample>?funid=0   94.61 ms -> 16.22 ms   -83%

diagram_file re-reads the report rather than being handed the page's copy,
which is sound because the renderer only ever looks at the unfiltered
function_matches, library_matches and reference_sample_entry - and that is
asserted rather than assumed: every diagram the pages display is byte-identical
before and after, and a test renders the page-filtered and unfiltered reports
and compares bytes, so a future mcrit that makes the renderer filter-sensitive
fails here instead of silently drawing the wrong picture.

The pagination count aggregated the whole report and threw it away, only for
the template to build it again for the page slice. getAggregatedFunctionMatches
groups by function_id and drops no group, so the count is the number of
distinct function_ids - asserted across three reports and six narrowings, and
those assertions pass on master too, because they prove the claim rather than
the change.

    matches_for_sample     1913 matches, 756 distinct   2.433 ms -> 0.033 ms
    matches_for_query      2072 matches, 537 distinct   2.372 ms -> 0.034 ms

load_cached_result matched `job_id in filename` against a name that also
carries a timestamp, so load_cached_result(app, "2026") returned some arbitrary
other job's report - and, having no reason to stop, it json.load()ed every
match and kept the last. Matching the id against the part of the name it
actually occupies and reading only the newest hit fixes both:

    the same job cached 20 times: 97.34 ms -> 1.45 ms

I did not use glob() for this. I measured it, and at 20000 cached files it was
slower than the scan it would have replaced.

And the jobs pages re-fetched the same family once per job carrying it, with
no `not in` guard - unlike the sample loop two lines above, which has one.
25 jobs in one family: 25 getFamily calls -> 1.

Two pre-existing hazards fixed on the way, because this change makes both more
likely to be hit: image.save() and the result cache both wrote in place while
other requests were reading them, so a reader could be served - and
browser-cache - a truncated PNG. Both now render to a temporary file and
os.replace it into position. And a job id from the URL is validated before it
is pasted into a cache path.
pytest and ruff used to arrive as transitive dependencies of mcrit and no
longer do, so a clean checkout could not run `make test` or `make lint`.
Install them explicitly in both the workflow and the target that claims to
set the environment up.
Issue #80. The reported symptom - a YARA rule truncated to ~120 characters
when copied - does not reproduce: a 5613-character rule copies intact. Two
other defects in the same eight lines do.

copyElementToClipboard built a detached textarea from $(element).html(). That
is innerHTML, i.e. the markup the server rendered, so a rule the user had
edited in the textarea was silently discarded and copied in its original
form - and Jinja's escaping of & < > came back as &amp; &lt; &gt;, which does
not compile as YARA. The element passed in is already the textarea, so the fix
is to read .value from it. Renamed to copyTextAreaToClipboard, since the old
name is what invited .html().

While there: it prefers navigator.clipboard.writeText but keeps the
select+execCommand fallback, because navigator.clipboard is not exposed on the
plain-HTTP deployments this project ships for; it restores the caret; and it
reports a failure instead of always claiming success.

The statistics table gains a Version column. The report carries none -
by_sample_id is block counts and nothing else - but a family job needs no extra
request to find it: getFamily already answers with the family's samples and the
caller is holding that entry. Only what the family did not supply is fetched by
id, which for a job naming samples directly is one call per row. A sample the
backend no longer has renders blank rather than failing the report.

Its columns are sortable, client-side, over a table that is already fully in
memory - so no round trip and no new query parameter to validate. Each numeric
cell carries its raw integer in data-sort-value, so "2844 (66.76%)" still sorts
as a number rather than as text. The sort symbols reuse the fa-sort ones
cursor_pagination_sort_symbol already renders. DataTables was not used.

Two sub-items of the issue needed no code. Sample coverage next to the
picblockhash is already there - result_unique_blocks.html renders
"{{ block["samples"]|length }} / {{ statistics["num_samples"] }}" and
renderRule writes "coverage: 3/3 samples" into every block comment.
Diversifying the block selection across function_ids belongs in mcrit:
generateBlockCover sorts on (uncovered-sample count, block score) and never
looks at function_id, and that is selection semantics.

Also removed a dead guard: {% if family_id is not none %} around the Samples
header, where family_id is never passed to render_template and Jinja evaluates
"Undefined is not none" as True - so the column always rendered and the guard
said nothing.
pytest and ruff used to arrive as transitive dependencies of mcrit and no
longer do, so a clean checkout could not run `make test` or `make lint`.
Install them explicitly in both the workflow and the target that claims to
set the environment up.
pytest and ruff used to arrive as transitive dependencies of mcrit and no
longer do, so a clean checkout could not run `make test` or `make lint`.
Install them explicitly in both the workflow and the target that claims to
set the environment up.
…gits

The tooltip is the score to two decimals, so it is not the source value. A
score of 2.501 renders as tooltip "2.50" and cell "3", and the assertion
shown == round(tooltip) would have failed on a correct page - a spurious
failure waiting for the next fixture regeneration, since these come from live
reports.

What the tooltip does pin is an interval: the score is within 0.005 of it. So
the cell has to be within half a unit of that interval. Truncation is still
caught, because it is off by the whole fractional part and these reports carry
plenty above 0.505 - 85.88, 82.70, 42.81, 60.99. Mutation-checked: restoring
%3d fails all three parametrisations.

Found by Codex on the PR.
Issue #42 asks for three things. (a) a working "Reset order" turned out to be
already implemented - dropping `?custom=` is the reset, and the view falls back
to the backend's `clustered_sequence` - so it is only covered by tests here, not
rebuilt. (b) and (c) are new: order the matrix by ascending sample ID, or by
family name and version.

The orderings are named (`?order=clustered|sample_id|family`) and resolved
server-side rather than frozen into a `?custom=` id list. A cross compare over a
couple of thousand samples would make that list several kilobytes of query
string, and a named order stays correct after a family is renamed. `?custom=`
still wins over `?order=`, so existing links keep working and the drag-and-drop
handler needs no change: it reads the sample list, which renders in the active
order, so a named order followed by a drag chains correctly.

One order applies to all six matrices, as before; the page copy now says so
instead of leaving it to be discovered.

Two things the review turned up and this fixes. `family_sort_key` coerces the
family and version fields with `str(... or "")` - both come out of an analysed
binary, and a backend that stored a non-string there took the whole result page
down with an `AttributeError`. And the invalid-`?custom=` path passed
`job_info=result_json` to `result_corrupted.html`, which rendered "Results for
Job:  are corrupted" and a Delete button posting to `/data/jobs//delete`; every
other corrupted render in the file passes `job_info`.

`?order=` is whitelisted through `parse_choice_query_param`, matching how the
other query parameters in this application are parsed.

+37 tests: 17 page-level and 20 unit-level. Four of the page tests fail before
this change.
Three corrections a review turned up.

The `{% if %}` around the label cell emitted a blank line either side and
re-indented the `<td>` on *every* page, including unfiltered ones - so the claim
that an unfiltered page is unchanged down to the rendered string was false. 27
rendered pages against master: 27 differed before, 10 after, and those 10 are the
pages this change is actually about (famid, samid, funid, exclude_library). All
three unfiltered pages are now byte-identical to master again. Fixed with
whitespace control.

The comment on `is_filtered` said "a score or library filter narrows the page
just as much". Only half true, and the half that is false is the interesting
one. `applyFilterValues` splits into family/sample filters and function filters,
and only the second group touches `filtered_function_matches`. Measured on the
captured 1-vs-N report:

    filter_direct_min_score=20         3/11 samples   1913/1913 functions
    filter_unique_only                 7/11           1913/1913
    filter_family_name=win.citadel     2/11           1913/1913
    filter_exclude_library             6/11           1812/1913
    filter_exclude_pic                11/11            576/1913
    filter_func_unique                11/11            557/1913

The six that narrow only the sample list leave every function match in place, so
the statistics and the function table below them stay in step - both showing the
whole job's function matches. They do disagree with the *sample* table, which is
a real gap, but not one this table closes by relabelling: the numbers would still
be the ones it is showing. A parametrised test now pins all nine, because a
comment cannot go red.
The claim that no other integer rendering was on a score was wrong.
`MatchedFunctionEntry.matched_score` is a float, and `MatchingResult` maxes it
into the `best_score` the aggregated tables show, so six more cells truncated
exactly as the sample columns did - one of them the "Best Score" column, which
is in the default `UserColumnSettings` active list and therefore renders on the
bare result URL with no query parameters at all:

    matches_for_sample     result_compare_all.html   100 cells, 11 truncated  93.75->"93"
    matches_for_query      result_compare_all.html   100 cells, 15 truncated  76.5625->"76"
    matches_for_sample_vs  result_compare_vs.html    100 cells, 14 truncated  90.625->"90"

Across the fixture that is 203/1913, 235/2072 and 56/468 function-match scores
losing half a point or more - issue #7's own complaint, one table lower on the
same page.

The existing test could not see them: its locator requires a `Percent: NN.NN%`
tooltip and these cells have none. The new test finds the column by its header
with an HTMLParser and reads the exact value off the render context through
Flask's `template_rendered` signal, so it asserts `int(cell) == round(score)`
rather than a tolerance.

And the existing test rendered only two of the four templates this change
touches: `?famid=` and `?samid=` were never set, so `result_compare_family.html`
and `result_compare_sample.html` - 8 of the 20 already-changed cells - were
untested. The parametrisation is now (report, query, template) triples and both
tests assert which template the request actually reached, which took two
corrections against the fixtures: `?famid=` has to go on the 1-vs-N report
(the 1-vs-1 family page has an empty sample table), and `?samid=`/`?funid=` have
to go on the 1-vs-1 report (the by-id pool is only complete there).

Mutating each of the five table cells back to `%d` individually now fails
between one and two tests each; so does each of the four already-shipped cells
on the newly reached templates.

242 -> 250.
Four findings from a review of this change, plus the smells it turned up.

**The cached files silently became mode 0600.** `tempfile.mkstemp` creates at
0600 by design - it is built for secrets - and `os.replace` preserves that,
where master's `open(path, "w")` produced the umask's answer. Measured on a real
render: the report JSON and both diagrams came out 0o600 against a 0o644
reference. Any deployment where nginx serves `instance/cache` directly, or where
a different uid reads it, breaks.

Fixed by not using `mkstemp`: the file is opened through `open(..., opener=...)`
asking for mode 0666, which `open(2)` masks with the umask in the kernel - so it
reproduces the in-place write exactly, with no `os.umask(0)` window to race and
no chmod. Measured: umask 022 -> 0644, 027 -> 0640, 002 -> 0664. A hardcoded
0644 was rejected because it would *widen* a hardened deployment. The opener also
adds `O_EXCL`, and using `open` rather than `os.open` + `os.fdopen` closes an fd
leak the previous version had on the `fdopen` failure path.

**`write_atomically` was completely untested.** Reverting both call sites to
master's in-place writes left the whole suite green, and the two tests named for
atomicity could not see it: one monkeypatched `renderStackedDiagram` to raise -
i.e. *before any write happens* - and the other asserted a file exists and
parses, which is true in place too. Both are replaced by tests that fail *during*
the write: a `HalfWrittenImage` that emits the PNG magic, flushes, then raises,
and a payload whose non-serialisable member makes `json.dump` emit ~50 bytes and
then raise. Against the in-place revert the diagram one now fails with
`assert 200 == 404` - the truncated PNG was actually served, which is the hazard.
Neither old body was deleted; both were kept under names that say what they
really test.

**The temp files sat in the directory the app serves.** `mkstemp(dir=cache_path)`
put them under `cache/diagrams`, which `diagram_file` serves for any `<path:filename>` -
a leftover `.incomplete-*.part` came back `200` with its partial content. They now
go to `instance/cache/incomplete/`, a sibling of the two cache directories. Chosen
over rejecting the `.part` suffix in the route because the hazard is "an unfinished
file is reachable", not "its extension is .part". Six traversal spellings checked
against the route, all 404.

**A non-canonical filter id did unbounded uncacheable work.** `DIAGRAM_FILENAME_RE`
accepted `-?\d{1,18}`, but the file written is named from `int(filter_id)`, so
`-famid_0001` rendered `famid_1` and then served 404 for the name asked for -
repeating the report parse on every hit, forever, behind nothing but
`@visitor_required`. The grammar is now `0|-?[1-9][0-9]{0,17}`, exactly how
`str(int)` spells it, which makes filename -> (job, filter) injective so the
on-disk short-circuit can always answer. 78.8ms and a `MatchingResult.fromDict`
per hit becomes 1.5ms and none. All four templates emit `'%d' % famid`, so
nothing the app produces changes.

**`load_cached_result(app, job_id)` logged through `current_app`** on its
unreadable-file path, despite taking `app` explicitly so it would not need a
request context. `RuntimeError: Working outside of application context.`

Also: two of the four parametrisations in
`test_the_diagram_does_not_depend_on_the_filtering_the_page_applies` were
vacuous - `{}` and `filtered_function_id: 0` narrow nothing, so both sides were
trivially equal. A second axis over user filters takes the mutation from 2/4
caught to 5/8. And the stale docstring claiming a job id is "pasted into a glob
pattern" - there is no glob anywhere in `mcritweb/` - now says the metacharacters
are there as a shape a reader expects refused.

326 -> 345.
Both things this change exists to do could be reverted with the suite green.

Restoring `copyTextAreaToClipboard` to the `$(element).html()`-into-a-detached-
textarea implementation that issue #80 is *about*: 245 passed. Deleting the entire
client-side sorting script: 245 passed. The sort test asserted three substrings -
`data-sort="number"`, `data-sort="text"`, `data-sort-value="2844"` - which is
markup, and its docstring claimed "sorting happens in the page" while nothing
verified that.

Both now have a lint that always runs and a browser test that drives the
behaviour, because either alone is insufficient: a lint never executes the script,
and a browser test leaves CI with no cover at all, since playwright is not a
dependency here.

The lint checks that the icon's onclick names the helper *and* that the id it
names exists (a helper wired to the wrong selector is as broken as none), that the
helper reads `textarea.value`, and that its code carries none of
`copyElementToClipboard`, `.html()` or `.innerHTML` - comments stripped first,
because the helper's own comment names the old implementation.

`tests/testBrowser.py` boots the same offline harness the suite already uses on a
loopback port and drives Chromium. The clipboard test *edits* the rule to
`rule edited { condition: 1 < 2 and 3 > 2 /* R&D */ }` before clicking, so the
assertion is about the current value rather than the rendered one; under the old
implementation the clipboard carries the shipped rule instead of the edit. The sort
tests read `textContent`, never `dataset.sortValue`, so they are not asserting the
implementation back at itself. It skips cleanly where playwright or Chromium is
absent - both paths verified, exit 0 either way.

And the "a family job needs no extra request" claim was dead in the suite.
`tests/fixtures/families.json` was captured off `/families`, which cannot carry
samples - `MinHashIndex.getFamilies()` reads storage, and only
`FamilyResource.on_get` fills them in - so `FamilyEntry.samples` was None in every
integration test and the shortcut never ran:

    Counter({'getSampleById': 3, 'getJobData': 1, 'getResultForJob': 1, 'getFamily': 1})

`regenerate.py` now captures `/families/{id}?with_samples=true` per family, the
fixture is rebuilt to match, and the fake honours `with_samples` so the two
endpoints stay distinguishable. The claim is asserted on the rendered page:
`getFamily == 1`, `getSampleById == 0`, and a version still present so "no
requests" cannot mean "no versions".

That fixture change made `test_unique_blocks_page_survives_a_backend_that_lost_a_sample`
vacuous - it stubs `getSampleById -> None`, which is no longer called on the family
path. Rewritten rather than deleted: it now drops a sample from the family so the
by-id lookup is reached, and asserts that sample's version is absent while the
others are present.

Six mutations, none surviving: the old clipboard implementation (2 failed), the
sorting script deleted (3), the helper renamed but still copying innerHTML (2),
the family shortcut removed (3), the fixture recaptured off `/families` (3), and
the fake serving one shape from both endpoints (1).

245 -> 251.
write_atomically opens the report cache in text mode, and that cache is served
back verbatim as the raw result - so what lands on disk has to be byte-for-byte
what json.dump wrote. Text mode does not promise that: it rewrites the newline
on any platform whose line ending is not LF, which on Windows makes every one a
CRLF and the served file stop matching the bytes it claims to be.

newline="" for the text modes; it is not a valid argument in binary mode, hence
the conditional.

Found by merging this branch with the raw-result download (#75), which compares
the served bytes against the backend's and fails on the difference. Neither
branch can see it alone - this one writes the cache but nothing here checks its
bytes, and that one checks the bytes but writes in place.

The new test is honest about its own limit: on Linux, where text mode translates
nothing, it passes with or without the fix, so CI cannot see the regression it
guards. Mutation-checked where it does bite - dropping newline="" fails it.
The statistics table already followed `filtered_function_matches`, which covers
`?famid=`, `?samid=`, `?funid=` and the three function-level filters. It did not
cover the six filters above the sample table: `applyFilterValues()` keeps two lists
and those six rebind only `filtered_sample_matches`, so the table stayed job-wide.

That left the same view reachable two ways with two different answers. On the
captured 1-vs-corpus report, `?famid=3` and `filter_family_name=win.dridex` show the
identical pair of win.dridex samples; the first said 4 own functions / 249.0 bytes
and the second said the whole job's 756 / 151654.0.

So define "on screen" as the intersection: a function match counts when its own
function survived the function-level filters *and* the sample it matched is still in
the sample table. That is the definition `filterToFamilyId`/`filterToSampleId`
already imply, since they narrow both lists in lockstep, and it costs nothing
unfiltered - `_aggregateMatchSampleSummary` derives the report's sample list from the
same function matches, so no `matched_sample_id` is ever missing from it. The
equivalence with the backend's own aggregation is unchanged and still asserted for
every captured report.

The trade is stated in the helper: for those six filters the statistics now describe
less than the function table below them, which mcrit still renders job-wide. The
page's subject is the samples it is willing to show, so that is the honest direction.

`test_which_filters_the_statistics_follow` is rewritten rather than dropped. It
recorded which list each filter narrows and asserted the statistics ignored six of
them, which is the behaviour this commit removes; it now records the same split and
asserts the statistics follow either way. Its indicator field moves from
`num_own_functions_matched` to `num_foreign_functions_matched`, because a foreign
function belongs to exactly one matched sample while an own function usually matches
in several, so the own-function count can sit still through a real narrowing.

Also pins `?samid=` at page level, which the PR had recorded as untestable offline.
It is untestable on the 1-vs-corpus and query reports, where `assign_matched_offsets`
reaches past the captured pool - but a 1-vs-1 report matched one sample, so the pool
that is complete for the report is complete for its `?samid=` page too.
tests/fixtures/README.md now says which of the filtered views render and which do not.
Two of the YARA half of issue #80 were still open.

The Block column of the unique blocks table is YARA syntax and is there to be
copied out, but it was wrapped by breaking every 80th character. That lands
mid-byte far more often than not: of the 51 sequences in the captured report
long enough to wrap, 41 were cut inside a token and 23 of those left an
odd-length run - "6a33" arriving as "6a3" and "3" - which no YARA compiler will
take. Short sequences never wrapped, which is why the issue saw it only on
"extensively long yara strings". It now wraps between bytes with mcrit's own
wrap_string, the way renderRule already wraps the rule itself.

The rule names the picblockhash and its sample coverage but not the function the
block sits in. mcrit builds the cover greedily by how many uncovered samples a
block adds, tie-broken on score, with no notion of function identity, so nothing
stops a rule from fingerprinting a single function - and `condition: 7 of them`
fails the moment that function is recompiled. The captured report spreads its
ten blocks over seven functions, so it is not degenerate here, but a reader
cannot tell. Of the two things the issue offers - include function_id, or make
the cover prefer a variety of them - this takes the first: which blocks to pick
is mcrit's decision to change, and forcing variety here would mean
reimplementing its set cover and trading sample coverage away for resilience
this data cannot calibrate. Naming the function costs no coverage and makes the
spread readable before the rule ships.

Getting at the cover means calling generateBlockCover and renderRule instead of
generateYaraRule, which is those two lines and then discards the cover. The
comment is rebuilt exactly as renderRule writes it and replaced by name, so an
upstream format change leaves the rule as it was rather than mangling it.
Rounding the columns fixed a contributor to "too far from the expected value"
but not the verification the issue asks for. The arithmetic is in mcrit, but the
captured reports carry both its inputs (matches.functions: every function with
its byte weight, instruction count and match list) and its outputs
(matches.samples[*].matched.percent), so it is checkable from here.

An independent transcription of _aggregateMatchSampleSummary reproduces all 150
shipped percentage cells across the three fixtures exactly. The calculation is
correct. What is wrong is around it, and the largest piece is ours: all twenty
score tooltips render "Bytes: X / binweight" above "Percent: Z%" where Z is not
X/binweight - the percent divides by matchable bytes, or for the nonlib columns
by matchable minus library. On the top match that offers the reader 84.28 and
states 85.79. The gap scales with the sample's library share, so it is unbounded,
where truncation was off by at most 0.994 across those same 150 cells. Truncation
therefore cannot have been the whole answer.

Also found, in mcrit, reported not fixed: MatchedFunctionEntry stores flag bits
under boolean names, so getMatchTuple maps flags 2 -> 4 and 7 -> 21 and a
toDict/fromDict round trip turns a pichash match into a library one. Latent -
MatchingResult.toDict has no production caller in either repo - but it does
explain our own standing TODO at MatchReportRenderer.py:249, since a pichash
match scores 100 and match_is_pichash contributes 2.

The test is a characterisation test, not a regression guard: it passes before and
after, and what it catches is a re-captured fixture that no longer agrees with the
formula the ADR documents. Five mutations of the transcription - wrong denominator,
no family adjustment, > for >= at the size threshold, nonlib sharing the plain
denominator, and dropping the score factor - each fail it.

250 -> 256.
Two of the three boxes issue #51 opens with, the search box having been done
already.

"sort should be done in mcrit and not only consider displayed elements": the
page's only sorting apparatus was a DataTable configured against '#job-table'
while the table is rendered with id="{{ active }}", so it never initialised, and
had it initialised it would have reordered the 25 rows on screen and left the
rest of the queue alone - the complaint verbatim. It is gone, replaced by a link
on the '#' column that flips `ascending`, which mcrit does apply to the whole
queue: mongoqueue's get_jobs sorts by _id before it skips and limits. That is
also all mcrit can do - nothing from getQueueData down to the collection accepts
a sort key - so Type, Started, Finished and Progress stay honestly inert rather
than offering an order that stops at the page boundary. Unlike the search, this
costs nothing: the browse path still fetches one page.

"what about cross jobs?" had a concrete answer rather than a design question. A
cross compare runs as one getMatchesForSampleVsGroup job per sample plus a
combineMatchesToCross job that merges them, and mcrit's Job.method_types - which
is where this page's category menu comes from - lists the group jobs neither
under "matching" nor in "all". So the jobs a cross compare does its work in had
no tab: they could not be browsed, ordered or searched, and the "Matching" count
left them out while the totals row above it counted them. Both now include them.

The order link carries only the five parameters the page understands. Built from
request.args wholesale it would hand url_for its own reserved keys straight from
the query string, and it drops the page number, since page 3 of one order is an
unrelated slice of the other.

New tests/testJobSort.py, eight cases, all eight failing before this change.
ADR-0003 established that the arithmetic behind the nonlib frequency score is
correct. #7 reports a value too far from the *expected* value, so the expectation
is what was wrong - and the hover text is what sets it. All twenty score tooltips
printed

    Bytes: 130682.88 / 155065          <- reference_sample_entry.binweight
    Percent: 85.79%

where the percentage is taken against a different total: matchable bytes (153466)
for the plain columns, matchable minus library-matching functions (152337) for the
nonlib_ ones. Neither is the binweight, so the reader was offered 84.28 and told
85.79. The gap scales as binweight/(matchable-library): 1.5 points here at a 0.7%
library share, and unbounded as that share grows.

Neither total is on the wire and neither can be recomputed here - the function list
carries no instruction counts and omits unmatched functions. But the divisor is a
property of the reference sample, identical on every row, so it does not need
recomputing, only inverting: 100 * matched_bytes_X / matched_percent_X recovers it
exactly from any row that scored above zero. recover_score_divisors() does that once
per render, off the unfiltered matches so a ?famid=/?samid= page agrees with the
unfiltered one, and the page duplicates none of mcrit's threshold logic. A group
where every row scored zero cannot be inverted; it falls back, and a zero numerator
reads as 0% over any total, so no cell is left inconsistent with its neighbours.

Mutation-checked per cell, not just in bulk. Reverting all twenty fails all five
pages; reverting one column across the four templates fails all five; reverting one
template fails exactly the pages that reach it - all 8 cells of
result_compare_all.html fails 2, and family, sample and vs fail 1 each.

Also does the browser walk AGENTS.md asks for and the branch had skipped.
tests/testBrowser.py drives Chromium over the four templates, because the offline
tests read HTML as text and so cannot see the two things that only exist once a
browser has the page: &#10; has to become a newline in the attribute, and hint.css
draws the tooltip from content: attr(data-hint), which is in no markup. Both checked,
plus no console errors. It skips where playwright is absent, which includes CI -
a browser download on every job of a four-version matrix is not worth one hover
text, and AGENTS.md now says so and says to run it locally.

The two mcrit defects in ADR-0003 stay reported, not fixed.

256 -> 263.
fix/72 holds 0003 and several branches in this round each wrote one. Different
filenames, so git merges them all without conflicting and the tree ends up with
ADRs sharing a number.
danielplohmann and others added 9 commits October 19, 2026 13:43
Resolutions:
- __init__.py: master's client imports and #142's matching_statistics; both the
  search-highlighting filters (#136) and #142's template global.
- result_compare_*.html: #244's heading levels with #142's recomputed statistics.
- tests/fixtures/README.md: #142's account of which filtered views the pools cover,
  then master's cfg_loop_functions.json section.
Every id in ?custom= becomes a row and a column of each matrix, and the lazy panes
(#213) carry the same rows as JSON, so one id repeated 200 times already rendered a
12.5 MB page, and an 8 KB request line would ask for millions of cells. order_samples
(#145) now refuses a repeated id, which shows the existing "cannot produce the chosen
custom ordering" page. #159's page text no longer claims the clustered order is
shared by all matrices: mcrit clusters each matching method separately.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
Resolutions:
- views/data.py jobs and job_by_id, __init__.py index, views/explore.py: master's
  batched lookups (#191) and #145's family guard, each over #115's describable_jobs,
  so a job whose payload cannot be read loses its own lookups, not the page. The
  index and the job overview now apply that guard to the batched calls as well.
- jobs.html: #115's no-match message for a search; otherwise the table with #115's
  order toggle and #119's per-category empty states.
- __init__.py / utility.py: imports and template globals from both sides.
- the job list tests follow #132's redirect to the named tab, which keeps Search,
  ascending and p in the URL;
- the malformed-payload test backend answers the batched getSamplesByIds and
  getFamiliesByIds with nothing, so the views fall back to the single calls;
- the corpus fake selects sample_ids on payload.descriptor as mcrit's mongoqueue
  does, instead of parsing every job, so an unreadable job still reaches the listing
  the test is about rather than failing inside the fake.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
… matched

#145 moved the diagram to its own GET, which checked only that a family or sample
exists. Any visitor could then render one PNG per family and per sample in the
corpus for every job, into a diagrams cache nothing bounds. The route now renders
a famid/samid diagram only for a family or sample the report's sample matches name,
which covers every filter the result page links to (library samples included), and
it no longer needs the isFamilyId/isSampleId round trips.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
Resolutions:
- views/analyze.py: master already moved each sample search to the top of its view;
  #174's copies at the old places are dropped and master's calls go through
  search_page. The hash-list redirect never rendered its search, so it gets none.
- __init__.py index and views/explore.py: master's batched lookups, guarded by
  #115's describable_jobs, with #174's typed entries for the searches.
- views/explore.py keeps the FamilyEntry import #174 dropped: #219's rename
  read-back (family_read_back) builds one.
Resolutions: master's mcrit>=1.12.0 pin and Python 3.12 floor stand (#212 raised
them to 1.9.0 and 3.11, which master has since passed); AGENTS.md names #212's
testMcritFloor.py beside master's pin; the changelog entry waits for the release;
data.py JOB_CATEGORIES carries #115's getMatchesForSampleVsGroup and #212's three
repair jobs.
#115 filters the category it fetched, and turned a failed read into an empty list,
so the page reported "0 jobs matching" and "No job's parameters contain ..." for a
backend that never answered. It now flashes the failure and shows neither sentence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
…nd name their jobs once

- every scheduling button on the server page asks for confirmation and disables
  itself on submit: each forces a recalculation over the whole corpus, so a stray or
  double click queued it, or queued it twice;
- the six scheduling routes carry mcrit_server_required: McritClient lets a
  connection error propagate, so an unreachable backend was a 500, not the message
  the helper's docstring promised;
- the three repairs' empty states say an admin can start them from the server page,
  which #212 made true;
- the job list names them through the job_name filter like every other job, so it
  and the job pages agree ("Repair stale MinHashes", as on the button).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG
@danielplohmann
danielplohmann deleted the branch integrate/ui-C1 October 19, 2026 20:14
@danielplohmann
danielplohmann changed the base branch from integrate/ui-C1 to master October 19, 2026 20:16
@danielplohmann
danielplohmann merged commit 6564602 into master Oct 19, 2026
12 checks passed
@danielplohmann
danielplohmann deleted the integrate/ui-C2 branch October 19, 2026 20:48
r0ny123 added a commit to r0ny123/mcritweb that referenced this pull request Oct 20, 2026
The entries for familiary#298 and familiary#299, from their change lists and their notes
for deployment, and familiary#212's entry, which was held back for this. familiary#212's
floor change is not repeated: master's mcrit>=1.12.0 was already in 1.6.0.
Fixes made during the two integrations to code that had not been released
are left out.
r0ny123 added a commit that referenced this pull request Oct 20, 2026
Conflicts in 23 templates and stylesheets, where batch C (#298, #299)
changed the same lines as this branch. Master's changes are kept and this
branch's are applied on top:

- Headings: #244's h2/h3 outline wins over this branch's restyled
  headings; the delete-job modal title is <h2 class="modal-title h5">.
- Job titles: master's job_name filter replaces job_info.parameters in
  the link hunt, maintenance and function result headings, keeping the
  breadcrumb this branch added.
- Empty states: master's empty_state macro replaces this branch's inline
  messages; the macro itself now uses a centred div instead of <center>,
  and so does the job search's no-match line.
- Cross compare: master renders one method's matrix and fills the other
  tabs on click. Its tab loop now shows this branch's labels, looked up
  by method name; the pane body is a div instead of <center>, and the
  script finds it by class.
- Admin page: the maintenance forms keep master's confirm-before-submit
  handler and this branch's spacing classes.
- Job errors: master's pre.job-error replaces the wrapper div.
- Row buttons: this branch's per-item aria-labels stay, with master's
  titles added where this branch had none.
- YARA copy: the button stays and calls master's copyTextAreaToClipboard;
  testResultPages accepts a button carrying that handler, not only an <i>.
r0ny123 pushed a commit to r0ny123/mcritweb that referenced this pull request Oct 30, 2026
Bumps setup.py and adds the version history entry, folding in the five
"unreleased" lines (familiary#263, familiary#262, familiary#261, familiary#293, familiary#254).

48 PRs since 1.6.0: batches C1 (familiary#298), C2 (familiary#299), security E (familiary#318),
UI F (familiary#339) and fixes G (familiary#340). No dependency changes.

The entry leads with the three things that affect existing deployments:
instance/ backups under WAL mode, the bounded result cache, and the
optional retirement of pre-1.5.0 API tokens.

Test count measured on the batch G tree, which master 30cbc8a is
identical to: 2428 passed, 19 skipped against mcrit 1.16.0, 2447
collected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants