Skip to content

Integrate batch F (UI): #303 #301 #305 #302 #293 #253 #254 #141 #130 - #339

Merged
danielplohmann merged 46 commits into
masterfrom
integrate/ui-F2
Oct 29, 2026
Merged

danielplohmann merged 46 commits into
masterfrom
integrate/ui-F2

Conversation

@danielplohmann

Copy link
Copy Markdown
Collaborator

Nine UI and usability PRs (batch F), integrated the same way as #318. Each PR's head commit is an ancestor of this branch, so GitHub marks them merged when this lands with a merge commit. #254 is stacked on #253's branch, and GitHub refuses to retarget stacked PRs, so it will stay open after the merge and gets closed by hand with a note.

PR change
#303 a Unique Blocks job's sample ids are folded into ranges with a count on the job pages, the jobs list and the result page. The job page's Task row opens into every sample, and the result page lists the requested samples by family
#301 the Statistics page shows the versions (MCRITweb, backend, client library, SMDA) and labelled Corpus, Index, Maintenance and escaper sections
#305 cross-compare picker rows from the URL can be clicked off; the users "All" tab; the jobs toggler; diagonal and zero-match matrix cells no longer link
#302 passes Lighthouse's td-has-header and target-size audits: labelled icon headers, a real header row in the cross matrix, id and family cells on one line
#293 the sample, family and per-family/per-sample searches filter as you type; 0x<offset> on a sample page finds the function
#253 UI/UX parity: leftover CSS removed, <center> replaced, shared breadcrumbs, consistent table headers, column settings Save/Reset fixed (400 on master), Admins-tab delete fixed
#254 pre-1.5.0 (32-character) API tokens are flagged on Settings and on the admin users page, with a bulk regenerate; ACCEPT_LEGACY_APITOKENS can retire them
#141 dump detection from the filename: a de-dump marker always wins, uppercase _0X addresses are read, 1-16 digit addresses, bitness 32/64 only
#130 a failed backend call on a single-object page answers with a 502/503 page, or a 404 naming what is missing, instead of "not found"

Integration

Built on master fe6b313 at the heads above. Conflicts and their resolutions:

One integration commit, 28ea8f5, keeps the FKIE logo in the navbar's top-right corner. #253 had turned it into a flex item that sat right next to the MCRIT logo. The logo is pinned top-right again as on master, at #253's 40px height, 3px from the top so it clears the menu's help icon by about 8px. It is only shown from 992px. @r0ny123, you may want to carry this into #253 if that branch is reused. The grey navbar background that #253 removes stays removed, as agreed.

Verified

🤖 Generated with Claude Code

https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8

r0ny123 added 30 commits October 2, 2026 17:33
…views

Comprehensive audit and normalization of the user-facing interface:

- Clean global stylesheet constraints in style.css (remove rigid min-width: 900px, html padding override, and restrictive form sizing), navbar.css, and signin.css.
- Semantic HTML landmarks and accessibility enhancements in base.html (<html lang="en">, <main class="content">, explicit logo dimensions for CLS prevention, aria-labels).
- Replace all deprecated <center> tags repo-wide with Bootstrap 5 flexbox/alignment.
- Migrate table headers across all views to standard Bootstrap 5 table-light.
- Standardize compare drill-down navigation with a shared compare_breadcrumb component.
- Fix broken controls: attach CSRF token to column settings AJAX/reset, fix duplicate IDs, properly close <i> icon tags in <th>, and fix orphaned for="block_count" label references.
- Redirect unfinished analyze.compare_submit_query route to analyze.query.
- Add automated regression guardrails in tests/testUiParity.py.
- Delete the now-unreachable compare_submit_query.html stub: the route
  renders a redirect to analyze.query since the previous commit.
- Apply .btn-clipboard to the clipboard button macro (a 24px touch
  target instead of the inline 20px style) and drop the unused
  .mcrit-dropzone rule from style.css.
- Give sortable column headers a keyboard handler to match the
  role="button" tabindex="0" they now carry.
- Add a meta description to base.html, closing the meta-description
  audit on #243; color-contrast in score-tinted rows and one-digit
  target sizes remain open there.
- Document deliberate changes the previous message omitted: the "View
  profile" dropdown items in the admin user lists linked to cross
  compare and no profile page exists, so they were removed; the
  nav-pill and accordion overrides in style.css were dropped in favour
  of Bootstrap defaults; .gitignore additionally covers .claude/ like
  CLAUDE.md.
The 32-character tokens issued before 1.5.0 stayed valid until their owner
happened to press regenerate, and nothing told anyone about them: an owner
could not tell a legacy token from a fresh one without knowing the length
rule, and an admin could not see how many were left without querying the
user table. The length is the discriminator - md5(uuid4().bytes) is always
32 hex characters, secrets.token_hex(32) always 64 - so no migration is
needed to find them.

- Tell the owner: the settings page warns next to a 32-character token that
  it predates 1.5.0 and should be regenerated, and highlights the
  regenerate button.
- Tell the admin: the users page counts the accounts still holding a legacy
  token, marks their rows with a key icon, and offers a "Regenerate all
  legacy tokens" action that rotates all of them at once. It confirms
  first, because rotating breaks whatever uses the tokens.
- Retire them: token_required answers a 32-character token with a 403 whose
  message says the token has the pre-1.5.0 form and must be regenerated -
  rather than the bare 403 an unknown token gets - once the operator sets
  ACCEPT_LEGACY_APITOKENS = False in instance/config.py. The default stays
  True for one release, so nothing changes until they choose the cutover.

Adds tests/testLegacyApitokens.py covering the discriminator, the gate in
both switch states, the settings and users-page notices, and the rotation
route; routePolicy gains the new admin route.
The row handler told a button click from a row click by event.target.id,
which only worked while style.css had `a > * { pointer-events: none; }`.
With that rule gone, a click on the edit pencil lands on the <i> icon, so
the row navigated to the family page instead of opening the edit dialog,
and the Malpedia icon opened its tab and also navigated the current one.

Skip any click inside a link or button, as the sample rows now do, and
drop the leftover console.log.
Issue #44 asks whether a filename containing "dedumped" should be treated as
unmapped. The answer is yes, and the code was already doing the opposite:
"dedumped" contains "dump", so request_filename_info classified it as a dump
and submit_or_query_dropzone.html clicked #dumped on that answer.

Since a de-dumped file has been *un*mapped, it normally carries no _0x... in
its name, so the user landed in "Dumped" with an empty base address and no
bitness. Submitting that prefilled form was a 500, in both submit paths:

    /analyze/query  options=dumped base_addr=""  -> 500
    /data/submit    options=dumped base_addr=""  -> 500   ValueError: invalid
                                                          literal for int()
    /data/submit    options=dumped, no bitness   -> 400   KeyError

Reproducing that turned up a third one the issue does not mention: int(x, 16)
accepts a leading minus, so a negative base address was accepted and forwarded
to the backend as base_address=-4194304. Now rejected.

Both fields are validated in params.py rather than inline, because the base
address is parsed at two call sites in two blueprints and one helper keeps
them from drifting. Neither defaults: guessing a base address for a memory
dump is a worse failure than an error message, and the dropzone's own error
handler reloads the page on a non-2xx - which is exactly what renders the
flash - so flash + 400 is the mechanism that actually tells the user.

The exclusion is the literal token rather than a (?<!de)dump lookbehind. The
lookbehind also suppresses "widedump" and "sidedump", and anchoring it with \b
does not help because _ is a word character in Python's re, which re-breaks
"sample_dedumped" - the exact case this is about. The token's one false
negative is a name carrying both "dedumped" and a real dump marker, which
costs one radio-button click.

The smda half of the issue belongs to that project, not this one.
base address the SMDA path never reads

Two findings from a review of this change.

`'dedumped' not in filename` is defeated by every separator and stem variant,
so issue #44's own symptom survived them. Measured through
`POST /data/request_filename_info` before the change: `sample_de-dumped.bin`,
`sample_de_dumped.bin`, `sample_dedump.bin` and `sample_dedumping.bin` all came
back `dump: True` with an empty base address - the dropzone clicks "Dumped",
prefills nothing, and the submit is refused. `sample_DeDumped.bin` came out
right only because `'dump' in filename` is *also* case-sensitive: two bugs
cancelling.

The rule is now: remove the de-dump markers, then ask what dump is left.

    DEDUMP_IN_FILENAME = re.compile(r"(?<![a-zA-Z])de[-_. ]?dump", re.IGNORECASE)
    DUMP_IN_FILENAME = re.compile(r"dump", re.IGNORECASE)

`\b` cannot work here - `_` is a word character to `re`, so `sample_dedumped` is
one token - and a strict token test would break `memdump`, which an existing test
pins as a dump. So only the de-dump marker is anchored, and on its left only:
"de" opens a de-dump where it starts a token and is the tail of the word before
it where it does not, which is what keeps `widedump` and `sidedump` dumps.
Subtracting rather than testing independently also fixes the false negative this
PR flagged and left: `dump_dedumped_0x400000.bin` is a dump again.

One deliberate behaviour change from master: `SAMPLE_DUMP.BIN` is now recognised.
The exclusion has to be case-insensitive to catch `DEdumped`, and leaving the
positive test case-sensitive would keep the cancelling asymmetry as a rule rather
than an accident. `MEMDUMP.BIN` is a real spelling, and a false positive costs one
radio click where the false negative costs filling the dump form by hand.

Separately, `is_dump_or_smda` gated the base-address and bitness validation, but
the SMDA path reads neither: `addReport(smda_report)` and
`requestMatchesForSmdaReport(...)` take no such parameter - checked in the
installed client, and the `else` branch in `data.submit` is unreachable because
`SmdaReport.fromDict` raises rather than returning falsy. So a valid SMDA report
was refused 400 on a field nobody reads, whenever the dropzone's header sniff
missed. Gated on the dump alone now.

That leaves the parse as the first thing an unreadable body reaches, so it gets
its own guard. Not a drive-by: the old 400 was accidental cover with a misleading
message, and the 500 behind it was already reachable on master whenever the
address happened to be filled - which the dropzone does for a `.smda` drop. Five
unreadable bodies across both routes now answer 400 and reach no backend call.

276 -> 315.
The search boxes of a sample's page, the family and sample lists and a
family's page now update their table once typing pauses. live_search.js
fetches the page the form would load and swaps in its table, pagination
and flashed messages; Enter still loads the page, and anything that is
not that page (an expired session, an error) loads it too.

The function list and the combined search page keep waiting for Enter:
an unscoped function search can take tens of seconds on the backend,
and aborting the request in the browser does not stop it (ADR-0016).

Following ADR-0014, the row clicks of the function, sample and family
tables and the edit modals' listeners are delegated, so rows swapped in
later behave like the ones loaded with the page, and the family-name
type-ahead also attaches to a modal on first focus.

On a sample's page a term that is only a hex number, like 0x6840, is
searched as offset:0x6840. It used to be matched against function names
and found nothing.
mcrit 1.15.0, which CI now installs, makes McritClient send through a
requests.Session() it opens in its constructor. The test swaps the
client's requests module for a stub with get and post only, so creating
the client raised AttributeError and all eight tests in the file
errored. The stub now hands itself back as the session. Same change as
in #285; it works with 1.13 and 1.15.
single_query_sample.html: master stopped printing "None" for a sample
without a timestamp; this branch replaced the row's valign attributes with
align-middle and closed the header cell with </th> instead of </td>. Kept
both.

testSampleTimestamp.py located that row by "Timestamp</td>", the
mismatched closing tag; it now accepts </th> as well, since what it tests
is the cell's value.
…y case

A de-dump tool that appends "_dedumped" to the dump's own name leaves the
dump marker and the base address in it. Dropping the de-dump marker and
asking what was left took foo_dump_0x00400000_dedumped for a fully
pre-filled dump, which would submit an unmapped file with is_dump and the
dump's base address. Any de-dump marker now means not a dump.

The dump marker became case-insensitive in 2bd571c, but the
address after it was still matched as "_0x" only, so FOO_DUMP_0X00400000
was a dump with no base address and no bitness, and its submit was
refused. The address is now read after "_0x" or "_0X".

The address also takes 1 to 16 digits and must end there: fewer than 8
digits used not to parse at all, and a longer run was cut to its first 16
digits. The bitness is still read off the digit count.
McritClient.addBinarySample leaves any other bitness out of the request,
and the server then disassembles the dump as 32 bit, so a 16 sent by hand
was dropped without a word. It is now refused like any other bad value.
Master brings #244, #136 and #119 into the pages this branch swaps.
- README: the unreleased lines this branch carried were folded into
  1.6.0; only the live search line stays above it.
- single_sample.html: #244's h2 headings and #119's empty message and
  #136's highlighting go inside the swapped region, next to this
  branch's form and status line.
- family_row.html, sample_row.html: #244 made the modal lookups
  null-safe with ?.; the delegated listener here already covers a page
  without the modal, and also a modal a search brings in, so it stays.
- testSubmitMetadata.py: master's fix from #297, in place of the port.
#119 replaced "No functions available" and "No samples available" with
messages per page; the region test now expects those. #136 marks the
search term in the rows; a new browser test types into the sample list
and checks that the rows swapped in carry the marks.
No conflicts. #174's search adapter now runs the sample page's function
search; the query it is given is still built by sample_function_query(),
so a bare hex number reaches the backend as offset:<term>.
Rebuilt on master after batch C, following the review: the default client
keeps answering None for every failure, so the lists, batch lookups and
annotations that leave the missing part out still do. A call site that
cannot carry on without the answer, or that says "not found" on a None,
goes through with_server_errors(), a copy of the client in mcrit's
raise_server_errors mode: a failure on the backend's side becomes a 502
page, and a None there means refused or not there. Where the view needs
the value, require_result() turns that None into a 404 page naming what
is missing. A requests exception becomes a 503 page, and a bare 502 or
504 on /api.

The result and link hunt pages no longer try to cache a report for a job
the backend did not return.
Conflicts in 23 templates and stylesheets, where batch C (#298, #299)
changed the same lines as this branch. Master's changes are kept and this
branch's are applied on top:

- Headings: #244's h2/h3 outline wins over this branch's restyled
  headings; the delete-job modal title is <h2 class="modal-title h5">.
- Job titles: master's job_name filter replaces job_info.parameters in
  the link hunt, maintenance and function result headings, keeping the
  breadcrumb this branch added.
- Empty states: master's empty_state macro replaces this branch's inline
  messages; the macro itself now uses a centred div instead of <center>,
  and so does the job search's no-match line.
- Cross compare: master renders one method's matrix and fills the other
  tabs on click. Its tab loop now shows this branch's labels, looked up
  by method name; the pane body is a div instead of <center>, and the
  script finds it by class.
- Admin page: the maintenance forms keep master's confirm-before-submit
  handler and this branch's spacing classes.
- Job errors: master's pre.job-error replaces the wrapper div.
- Row buttons: this branch's per-item aria-labels stay, with master's
  titles added where this branch had none.
- YARA copy: the button stays and calls master's copyTextAreaToClipboard;
  testResultPages accepts a button carrying that handler, not only an <i>.
Two CSS lines this branch touched kept trailing spaces, testUiParity.py had
three blank lines between two tests, and .gitignore gained an entry for a
local tool directory that master does not list.
The page printed the first twelve keys of the backend's status as they
came, raw names and all. On mcrit 1.13 that cut off the stale MinHash
and block hash counts, the rows that say a maintenance job is due, and
the versions of MCRITweb and of the backend were nowhere on it.

statistics_page() in mcritweb/statistics.py turns the status answer
into a Versions, a Corpus and an Index section with labels and
thousands separators, a Maintenance list that names a count only when
it is above zero together with the job that redoes it, and the escaper
fingerprints per architecture. A key it has no label for goes under
Other, so a newer backend is never trimmed. The versions are
MCRITweb's (MCRITWEB_VERSION), the backend's (backend_version, moved
from the admin view to views/client.py so both pages share it), the
installed mcrit client library's and the backend's SMDA.

The database's last change is shown as a UTC date and time, which is
the clock mcrit stamps it with on every release from 1.12 on.

The page keeps asking for the distinct PicHash count, which is one of
the corpus figures. A backend without a status answer still gets the
versions and a sentence saying so.
This branch put a static description into base.html's head, and #244
later gave master a {% block description %} for the same purpose. The
batch C merge kept both, so every page carried two descriptions and a
crawler could take either one. Master's block stays, since a page can
override it, and the static line goes.

testAccessibility's page check now reports more than one description,
which fails on all 35 rendered pages without this change.
Lighthouse's accessibility category still failed two audits on master.
On a phone, target-size cost the 1vN and compare-sample result pages 4
points: an id or family cell holds the value and the icon that filters
by it, and a narrow column wrapped them onto two 21.6px lines, closer
than the 24px targets WCAG 2.5.8 asks for. td-has-header failed on the
index and the compare-all and compare-family results, where a header
cell had no text: an empty <th> over a buttons column, or an icon whose
title is a tooltip rather than header text.

Every such header gets a visually-hidden label and its icon
aria-hidden; icons left open inside a header are closed, since an open
aria-hidden <i> would hide the label too. The cross-compare matrix had
a header row of <td>; it is <th> at normal weight and looks the same.
Id and family cells keep their content on one line, so a family name
with spaces takes the column's width instead of wrapping.

testAccessibility's page check now reports a header without text, a
table without a header cell, and a value and its filter link that may
wrap; testTableLayout pins the nowrap rule.
The Task row listed every sample id the job names, up to the 250 a
selection may name or a whole family's. Runs of three or more
consecutive ids now read as one range, followed by the count the range
hides, and past five ranges or single ids the row gives only the count.
The row's title still carries the call with every id. One id reads
"sample", not "samples".

Closes #300.
Dropping the tutorial's .content rules also dropped the 1em gap under
every field of the result pages' filter forms. At desktop width their
checkboxes are under 24px tall and sat flush against the field above
and the Filter button below, so Lighthouse's target-size audit failed
the 1vN result there. form.form-inline gets the gap back.

The sample Analyze buttons carried an aria-label that left out the job
count they show (label-content-name-mismatch, WCAG 2.5.3). They are
named by their content now: the count, visually hidden words and an
icon hidden from assistive technology.

testAccessibility's page check now fails on an aria-label that leaves
out a word the control shows, and testFilterFormTargetsBrowser measures
the spacing WCAG 2.5.8 asks of the filter forms in Chromium.
- cross_compare.html: the ids the URL carried in ?cache= were numbers while
  a clicked row's id is cell text, so a tinted row could not be clicked off
  and Add sent it twice. selected now holds strings.
- users.html: the "all" pane rendered as "fadeshow active", two panes ran
  their class into id, and aria-selected named "All users" on every tab
  URL. Each now follows the active tab.
- jobs.html: the navbar toggler named its menu with Bootstrap 4's
  data-target, which 5.0.2 does not read, so below 992px it opened nothing.
- result_cross.html: a pair without matches, the diagonal among them, linked
  to a filtered page that listed nothing and whose diagram 404ed. Such a
  cell now links nowhere.
…e too

The jobs list printed the job's sample count and then every id, and the
finished result page linked every id in its Requested Samples row. Both
now print the Task row's line through the same unique_blocks_samples,
registered as a template filter. The jobs list keeps the ids it used to
print in the line's title. On the result page the line opens, the way
linkhunt.html folds a long cluster, into every sample, still linked.
r0ny123 and others added 16 commits October 24, 2026 20:12
Unfolded, the Requested Samples row now gives each family on a line of
its own with its samples linked after it, in the order the job names
them, the way the jobs list groups a cross compare's. An id the backend
has no sample for goes under "-". The page already fetches every
requested sample in one batch for the versions in its statistics, so the
families come from entries it holds and cost no further request.
The job page keeps both: master's job id check before getJobData, and
this branch's with_server_errors around it.
The ids its line folds into ranges or a count were only in the row's title,
which touch and the keyboard cannot reach. The line is now the summary of a
<details> that lists every sample, linked, in the order the job names them;
a line that names each id already stays as it was. This is the Task row of
the job overview, in-progress and failed pages.

The jobs list keeps its title: a click anywhere on a row opens the job's page,
so an expander there would open and navigate at once, and that page is where
the row unfolds.
style.css's `.content form` rule, which this branch drops, gave the zone half the
column, 608px on a desktop. In the flex column that replaces <center> a form is as
wide as its content instead, and the zone's content is Dropzone's message: 44px wide
until Dropzone had attached, 239px after, short of the 600px max-width. Lighthouse
counted the widening as a layout shift wherever Dropzone attached after the first
paint.

Both zones are width: 100% up to 600px now, the import page's too, whose margin: 10%
never applied (.content form set 1em).
# Conflicts:
#	mcritweb/static/style.css
#	mcritweb/templates/cross_compare.html
#	mcritweb/templates/result_compare_all.html
#	mcritweb/templates/result_compare_family.html
#	mcritweb/templates/result_compare_sample.html
#	mcritweb/templates/result_compare_vs.html
#	mcritweb/templates/result_cross.html
#	mcritweb/templates/statistics.html
#	mcritweb/templates/table/family_row.html
#	mcritweb/templates/table/sample_row.html
#	mcritweb/templates/unique_blocks.html
#	tests/testAccessibility.py
# Conflicts:
#	README.md
# Conflicts:
#	mcritweb/views/explore.py
#253 turned .right-logo from an absolutely positioned corner logo into a
flex item, which put it right next to the MCRIT logo instead of at the
right edge. Pin it top-right again, as on master, at #253's 40px height.
The logo is only shown from 992px (d-lg-block), so narrower layouts are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
@danielplohmann
danielplohmann merged commit e7e75fa into master Oct 29, 2026
8 checks passed
@danielplohmann
danielplohmann deleted the integrate/ui-F2 branch October 29, 2026 19:46
r0ny123 pushed a commit to r0ny123/mcritweb that referenced this pull request Oct 30, 2026
Bumps setup.py and adds the version history entry, folding in the five
"unreleased" lines (familiary#263, familiary#262, familiary#261, familiary#293, familiary#254).

48 PRs since 1.6.0: batches C1 (familiary#298), C2 (familiary#299), security E (familiary#318),
UI F (familiary#339) and fixes G (familiary#340). No dependency changes.

The entry leads with the three things that affect existing deployments:
instance/ backups under WAL mode, the bounded result cache, and the
optional retirement of pre-1.5.0 API tokens.

Test count measured on the batch G tree, which master 30cbc8a is
identical to: 2428 passed, 19 skipped against mcrit 1.16.0, 2447
collected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants