Skip to content

Ask the server for a sample's jobs and download the family and sample lists after Convert - #39

Merged
danielplohmann merged 9 commits into
familiary:mainfrom
r0ny123:faster-server-downloads
Oct 15, 2026
Merged

danielplohmann merged 9 commits into
familiary:mainfrom
r0ny123:faster-server-downloads

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

This needs #30, #31, #33 and #35 to land first. The branch merges #30 into #35 (f9f28a3), and its own commits are 6de1a96 and 55d1ef0.

flowchart LR
  main --> 31["#31"] --> 33["#33"] --> 35["#35"] --> M((merge))
  main --> 30["#30 getQueueData job selectors"] --> M
  M --> this["this PR: faster server downloads"]
Loading

Three requests around Convert and "Fetch Matching Result" spent time on data the plugin throws away, or ran on the UI thread.

The job list downloaded every job on the server and filtered them in the plugin. On mcrit.malpedia.io that is 1,992 jobs to find the 2 that mention the sample. MCRIT 1.13 ignores the sample_ids selector that #30 added. Its filter is a case-insensitive substring match on the job's method and parameters, so the plugin now sends the sample id as the filter and keeps its exact match on the client side. The server's answer is a superset of what the client keeps. Every matching method in MCRIT 1.13 has "Matches" in its name, so the client's case-sensitive check for it keeps the same jobs as the old case-insensitive server filter.

Convert waited for the family and sample lists before it finished. They now download in the background after the report is built, side by side instead of one after the other, behind a lock so that one download serves every widget. The family chooser for an unknown sample waits for them, and Convert stays blocked until that dialog closes, so a second click cannot queue a second dialog. On IDA this is a no-op: the requests run synchronously as before, only in parallel.

Convert also asked the server whether the sample is known, on the UI thread, after the conversion finished. Against mcrit.malpedia.io that froze Binary Ninja for 1.4 to 1.9 s. The lookup now runs in the conversion's background work, _convertAndFetchRemote.

These are timings for sample e0b6f853… in Binary Ninja 6.0 against mcrit.malpedia.io, with #35 as the base:

step #35 this PR
Convert, until "Fetch Matching Result" is enabled 10.1 s 3.4–3.9 s
longest UI stall during Convert 1.4–1.9 s 0.13–0.14 s
family and sample lists present, from the start of Convert 10.4 s 5.9–8.7 s
job list request 2.6 s 1.6 s

The server's response times varied a lot between runs. The sample list request alone took between 3.7 and 8.8 s across runs of the same code, so the list timings are wide. Timed against the server without the GUI, the job list went from 1.45 s (1,992 jobs) to 0.9 s (2 jobs), where the rest is latency. The two lists went from 3.9 s one after the other to 2.4 s side by side.

getResultForJob(compact=True) was considered and not used. MCRIT 1.13 ignores it and returns the same 11 MB, and in the mcrit 1.9.0 source it would drop matches.functions, which the overview needs.

If starting a list download fails, Binary Ninja's run_request logs the error without calling back, and the chooser does not open. The list queries catch their own errors, so only a failure to start the thread leads there. Even then, the request clears the guard itself, so a second Convert can try again instead of every action staying blocked. For an unknown sample the lists are downloaded once, by the chooser's request. A second background download used to wait for it and repeat a failed one.

pytest gives 223 passed, run_quality_checks.py and the sync checks pass, and the Binary Ninja GUI test passes all 180 checks against a local MCRIT 1.12.0. The IDA tests were not run.

…ts after Convert

The job list for a sample sent every job of the server and filtered them in
the plugin. The sample id now goes to the server as the job filter, and the
exact test stays because the filter matches the id as text and older servers
ignore sample_ids. Against mcrit.malpedia.io this takes 1.45 s down to 0.9 s.

Convert waited for the family and sample lists. They now download in the
background after the report is built, side by side and once for every widget
that asks, which takes 3.9 s down to 2.4 s. The family chooser for an unknown
sample opens once they have arrived.
… once

After a conversion, _applyConvertedReport asked the server whether the
sample is known, on the UI thread. Against mcrit.malpedia.io that froze
Binary Ninja for 1.4 to 1.9 s on every Convert. The lookup now runs in
the conversion's background work, _convertAndFetchRemote, which already
existed for it. IDA runs that work synchronously behind its wait box as
before.

When the sample is unknown, the family dialog waits for the family list
to download, and a second Convert click in that time queued a second
dialog. Convert now stays blocked until the dialog closes.

Convert on sample e0b6f853... in Binary Ninja 6.0 against
mcrit.malpedia.io, until Fetch Matching Result is enabled:

  run      before: duration / longest UI stall   after
  1        4.05 s / 1.88 s                       3.46 s / 0.14 s
  2        3.51 s / 1.43 s                       3.52 s / 0.14 s
  3        3.52 s / 1.45 s                       3.41 s / 0.13 s
# Conflicts:
#	CHANGELOG.md
# Conflicts:
#	CHANGELOG.md
# Conflicts:
#	CHANGELOG.md
#	tests/core/test_mcrit_interface.py
…uest

On the first Convert of a sample the server does not know, the family
and sample lists were requested twice in a row: once in the background
and once for the family chooser. The second waited for the first and,
if the first had failed, downloaded both lists again, so against an
unreachable server the chooser opened only after a second timeout. The
background download now runs only when the chooser does not need the
lists.

_building, which keeps a second Convert from opening a second chooser,
was cleared only by the chooser. A request that raises never reaches it,
which left Convert, Export, Upload and the YARA builder answering that
the report was still being built until a restart. The chooser's request
now clears it when it raises.

tests/ui_qt/test_convert_flow.py covers both branches and the failure.
@danielplohmann

Copy link
Copy Markdown
Collaborator

#36, #38 and #46 are merged, so this conflicts with main now, in CHANGELOG.md only. Could you merge main in, keeping all the bullets with main's first as before? I'd like to take the open PRs one at a time in stack order: this one, then #44, then #41.

# Conflicts:
#	CHANGELOG.md

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The merge of main (4ef0f43) keeps all the changelog bullets with main's first and touches nothing else. I had read the PR before the merge and had nothing to add. Lint, the sync checks and the suite pass on main plus this branch (268 tests), and #44 and #45 merge cleanly on top of it.

The job-matching bug I found while reading it predates this PR and is filed as #47.

@danielplohmann
danielplohmann merged commit caa8cef into familiary:main Oct 15, 2026
10 checks passed
@r0ny123
r0ny123 deleted the faster-server-downloads branch October 15, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants