Repository navigation
Make the IDA plugin as responsive as the Binary Ninja one - #41
Conversation
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md # tests/binja/gui_integration.py # tests/core/test_mcrit_interface.py
# Conflicts: # CHANGELOG.md # mcrit_plugin/core/minimcrit/client/McritClient.py
# Conflicts: # CHANGELOG.md
IdaBackend.run_request ran every request on IDA's main thread, so a slow or unreachable server froze IDA for up to the request timeout, and fetching labels for a large result for about a minute. Requests now run on a pool of four threads. A main-thread ida_kernwin.register_timer hands each result to on_done, so workers never wait in execute_sync and no result is delivered while a dialog opened by another result is still open. Closing the MCRIT form cancels the queued requests and drops the results still to come; a failed request is reported in a warning named after it. The IDA GUI integration test waits for each action's result instead of expecting it synchronously.
A Function or Block Scope query that waits behind other requests while the cursor moves on is now skipped when a worker picks it up, instead of being sent and its answer dropped.
Convert asked the server whether it knows the sample inside the export work. On IDA that work runs on the main thread behind a wait box, so the wait box also waited for the server. The lookup now runs as a request once the report is built.
smda selects its ida_domain backend whenever that package is importable, and inside IDA its per-call wrappers make the export about twice as slow as smda's IDAPython backend. The IDA backend now installs a subclass of smda's IDAPython interface that reads instruction bytes with one get_item_size lookup and walks code references with ida_xref directly. The reports are byte-identical; in IDA 9.4 with ida_domain installed, a 2,268-function PE exports in 1.4 s instead of 3.1 s and a 3,668-function binary in 3.3 s instead of 6.9 s.
With use_smda_for_analysis on, Convert exported the disassembler's analysis only to print how its function set differs from SMDA's. Backend.get_function_offsets lists the functions the export would contain, skipping external functions as smda's exporter does, in milliseconds; the comparison now uses it and the export remains the fallback when SMDA's analysis yields no report.
Every cursor event refreshed the scope widgets, so scrolling through code sent a query per function. IDA's view hooks now refresh once the cursor has rested for 150 ms, as the Binary Ninja sidebar does, and look the view up again by title when the timer fires instead of keeping a handle to a view that may have closed.
A stale Block Scope query was skipped only before it started. One already running still looked up every block hash of its function, up to one request per block, while the cursor had moved on. The lookups that have not started yet now stop as soon as the cursor leaves the function; the skipped hashes stay uncached, so the next visit fetches them.
IDA's run_background ran its work synchronously behind a wait box, because IDA's API is main-thread only, so Convert, Upload and Export held IDA for the whole export. Disassembling and hashing, about two thirds of it, never touch IDA's API. run_background now runs its work on a worker like run_request. The backend methods that read the database run on the main thread through execute_sync with MFF_READ when called from a worker. The export reads everything smda's exporter asks for into SmdaInterfaceSnapshot on the main thread, behind a short wait box, and the exporter then runs against the snapshot on the worker. The reports are byte-identical with smda 4.3.10 and 4.9.0 on x86-64, arm64e and PE binaries. The main thread's share of the export is 26 to 35 percent, for example 1.0 s of 3.3 s for a 3,668-function binary.
The client's requests.Session kept urllib3's default of 10 connections per host. Block Scope's eight concurrent lookups plus the other requests running at the same time exceed that, and urllib3 then opens connections it discards afterwards, each with its own TLS handshake. The session now pools 16.
A worker's database read waits in execute_sync until IDA's main thread runs it. If the MCRIT form closes, possibly with its database, in between, the read now raises instead of reaching a database that may be gone.
Function Scope sends the server an outline of the local report holding only the queried function. The outline was built by serializing the whole report and dropping its functions. In IDA 9.4 against a 2,268-function sample, the first query after Convert held the main thread for 0.9 s for that, including a full garbage collection the allocations triggered. A shallow copy of the report with no functions serializes in 0.1 ms to the same outline, and leaves the session's report untouched.
Removed because nothing reaches them: - McritInterface.queryPicHashMatches and the session's pichash_matches, pichash_match_summaries and picblockhash_matches it alone wrote or read. - McritInterface.queryFunctionEntriesBySampleId and the session's remote_function_mapping: the mapping's only reader, McritSession.getFunctionInfos, had no callers, so the query was a server round trip the integration tests made without checking anything. - SampleInfoWidget's aggregation of a raw result dict, with its two helpers and McritSession.getSampleInfos: the session only ever holds a MatchingResult, which the first branch handles. - ScoreColorProvider.uniqueScoreToColor, OverviewTable.NUMERIC_COLUMNS, two unused test fakes, and commented-out code in the scope widgets. - MCRIT4IDA_PLUGIN_ONLY, a constant False no setting could change, and the branch of the script entry it guarded. - Nine icons no code loads.
Each chunk of the label fetch replaces the labeled entries, which invalidated the Function Overview's aggregation and rescanned every match. With 221,398 matches that took 0.4 to 1.1 s per chunk on IDA's main thread. The families, samples, functions and library matches of a function at a threshold do not depend on labels, so they are now kept per result and threshold. Labels are mapped to the local functions through an index of the matches by remote function, built once per result, so a label update and the label filter only touch the matches of labeled functions. On a synthetic result of that size a label update takes 10 ms instead of 110 to 140 ms, and in IDA the longest main-thread stall during the label fetch fell from 1.1 s to 0.4 s. The index holds tuples rather than lists, which the garbage collector stops tracking.
resizeColumnsToContents measures up to QHeaderView's resizeContentsPrecision rows, 1000 by default. For a function with 884 matches that took 179 ms of the 546 ms Function Scope needed to show them; with 100 rows it takes 26 ms. The tables list their best matches first, so the first rows are representative.
The match table was a QTableWidget with an item per cell, each created and colored on its own: 7,264 items for a function with 884 matches. In IDA 9.4 against a 2,268-function sample, showing them held the main thread for 0.47 to 0.55 s. MatchTable.py holds one row per match in a QAbstractTableModel, with the texts and sort values computed once and one color per score, and the view asks only for the cells in view; the same function now shows in 0.23 s. As before, numbers sort by value, the rows keep the order they were filled in until a header is clicked, and the chosen column survives a refill. Double clicking a match and copying its sample's SHA256 read the match behind the row instead of parsing cell texts, so they no longer need the function id or sample id columns in the table.
The block summary and the block match table were QTableWidgets with an item per cell. They now use MatchTable.py's model and view, generalized to rows of blocks as well as matches: 1,361 blocks fill in 30 ms instead of 51 ms and 2,000 block matches in 22 ms instead of 36 ms on Qt's offscreen platform. Offset and hash columns sort by value; as text, 0x10 sorted before 0x9. Selecting, jumping to and graphing a block, and copying a match's SHA256 read the block or match behind the row instead of parsing cell texts. The SHA256 copy raised before when the sample id column was not configured, and the other clicks did nothing without the offset or function id column. Rows without a tint keep the style's text color, as before, and the match at the selected block stays preselected after a sort.
The table of names from the matched functions was the last Function Scope table with an item per cell. It now uses MatchTable.py's model and view with one row per label. Double clicking a label's function id opens its graph and double clicking the label names the current function after it, as before, reading the label behind the row instead of parsing cell texts.
Both tables created an item per cell and measured each row's height on its own with resizeRowToContents. For a result with 4,071 matched samples over 399 families, the size of sample e0b6f853's, showing the result took 235 ms and clicking the family with 1,956 samples 194 ms on Qt's offscreen platform. With MatchTable.py's model, which gains an optional text alignment for the centered family table, they take 5 and 3 ms. Every cell is one line, so the default row height is the same, and every column stretches, which made sizing columns to their contents wasted work. Selecting a family reads the family behind the row instead of the text of the third column. A missing family, version or SHA256 shows as blank instead of failing the fill. The result chooser keeps its QTableWidget: it lists the matching jobs of one sample, usually a handful.
Three regressions of the table model ports: - Block Scope preselected the match at the selected block with selectRow, which Qt ignores while clicks select single cells, so nothing was selected. MatchTableView.selectEntireRow selects the row through the selection model. - Sample Info showed the first family in the user's sort order instead of the best matching family when its previous selection was gone, because the model sorts the row list in place; the best family is now taken before the rows reach the model. - A new table showed a descending sort arrow on its first column although the rows were in fill order. The arrow is now cleared until a header is clicked. Tests pin all three, and a refill test now uses rows the saved sort has to reorder.
The interpreter joins a ThreadPoolExecutor's threads before it exits, so quitting IDA while a request ran, a label chunk can take a minute, would wait for that request to finish. Four daemon threads now take the requests from a queue; closing the form stops them and drops the requests not yet started.
…e-block-coloring) # Conflicts: # CHANGELOG.md # mcrit_plugin/ui_qt/widgets/BlockMatchWidget.py
…ction-reuse) # Conflicts: # CHANGELOG.md
Brings in the 1.x upgrade note in the README; the changelog already has every bullet from main.
# Conflicts: # tests/ui_qt/test_function_overview.py
The test waited until the cache held server labels, but the automatic fetch after loading the result may have cached them already. The wait then ended while the clicked fetch was still running, and its closing activity line could replace the one Import labels writes, which a later check reads. The wait now also requires the fetch button to be enabled again.
Brings in the setup-uv bump; the changelog and the client already had everything from main.
…-coloring # Conflicts: # CHANGELOG.md
_label_free_aggregates, which a label chunk reuses so it only recomputes the labeled matches, was keyed by threshold and never trimmed, so stepping the spin box kept every threshold's per-function sets, like _aggregates did. It now keeps only the current threshold.
# Conflicts: # CHANGELOG.md # mcrit_plugin/ui_qt/widgets/BlockMatchWidget.py
The cursor debounce kept only the title of the last view that reported an event and refreshed that one when the cursor settled. A move to a new function in IDA View-A followed within 150 ms by an event from a hex view or the MCRIT graph viewer refreshed only that other view, whose current function is None, so the live queries skipped the move until the next event in the disassembly. The debounce now remembers each view that reported since the last refresh and refreshes them in order.
Block Scope looks up a function's block hashes on a thread pool inside its request, and the interpreter joins a ThreadPoolExecutor's threads at exit, running what is queued. Quitting IDA during a Block Scope query therefore still waited for all its lookups, which the move to daemon request threads meant to avoid. The lookups' stop check now also ends them once the backend is closed, which happens when the form closes; Backend.closed defaults to False for backends that never close. Also corrects a comment from before the export moved off IDA's main thread: only reading the database still runs there, behind a wait box.
# Conflicts: # CHANGELOG.md
|
#39, #44 and #45 are merged, so this is the last one in the stack. It needs
With the earlier PRs merged, the diff is down to your own commits, so I'll review it once |
- McritInterface: queryFunctionEntryById and querySampleEntryById, which main folded into queryRemoteFunction, and queryFunctionEntriesBySampleId, which this branch removed, have no callers left, so none of them stays. - BlockMatchWidget: the block match double click keeps reading its row from the table model and gains main's _graph_fetches guard: a repeat click while that function is being fetched is ignored, show() discards the id first, and the activity line reports the opened graph. The _shown_function/_shown_block tint state from main stays. - test_mcrit_client: main's client tests and the session pool test are both kept. - CHANGELOG: main's Unreleased bullets first, then this branch's own. Two of main's sentences said IDA still runs requests on its main thread, which this branch changes, so they go.
|
Lint, the sync checks and the suite (333 tests) pass. Against a local MCRIT 1.15.0 with SMDA 4.10.0, the IDA 9.4 GUI test (146 checks), the IDALib test and the Binary Ninja 6 GUI test (196 checks) pass. |
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks, this is a big step for IDA, and the merge of main (1db732e) is resolved as discussed. I read the 24 commits in three parts. Lint, the sync checks and the suite (333 tests) pass on the merged tree.
- IDA threading:
- Every backend method a worker reaches is wrapped in
_on_main_thread, andregister_timeris only called from the main thread. - The main thread never waits on a worker, so
execute_synccannot deadlock. SmdaInterfaceSnapshotcovers every read of smda's exporter, checked against 4.9.0 and theIdaExporterof 4.3.0.- Results after a close and database reads queued before it are dropped. The
_buildingguards are reset on every failure path.
- Every backend method a worker reaches is wrapped in
- Shared core:
- The vendored client keeps core 1.15.0's interface; the pool only mounts an adapter on the private session.
- The removed code has no callers left.
get_function_offsetsgives the same function set as the export it replaces.- The label-only recompute matched the old full recompute on 300 random results with changing labels, threshold and filter.
- Table models: every converted table behaves as before under PySide6 and PyQt5: text, sort keys (hex by value), colours, preselection, and click handlers mapped through sorts and resets. The cell handlers no longer depend on the configured columns either.
One thing before merge. ec9c9ea selects smda's IDAPython interface even where ida_domain is installed. That interface's _cbEnumImports builds module + "!" + name, and IDAPython passes no name for an import by ordinal that IDA cannot name. That raises TypeError, and Convert reports "exporting SMDA report failed". The ida_domain backend these users had before maps that case to "#<ordinal>" (IdaDomainInterface.py:154). The fix belongs in smda (danielplohmann/smda#398). Until a release has it, could FastIdaInterface override _cbEnumImports with the same fallback, name or f"#{ordinal}", plus a unit test with a nameless ordinal import?
Since this branch is a fork, CI could not run the licensed IDA job. I dispatched it on 1db732e from a branch of this repository: https://github.com/familiary/mcrit-plugin/actions/runs/38722186612. If the override lands, I'll run it again on the new head.
Convert now exports through smda's IDAPython interface even where the ida_domain package is installed. That interface's _cbEnumImports builds module + "!" + name, and IDAPython can pass no name for an import by ordinal that IDA cannot name, which raises TypeError and fails the export. smda's ida_domain backend, which these users had before, maps that case to "#<ordinal>". FastIdaInterface now passes the same fallback to smda's callback until a smda release fixes it (danielplohmann/smda#398).
|
Thanks for the review and for dispatching the licensed IDA job. df48778 adds the override:
Lint, the sync checks and the suite (334 tests) pass. |
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks. df48778 adds the fallback as asked: FastIdaInterface._cbEnumImports hands name or f"#{ordinal}" to smda's callback, with a test that fails without it and a changelog note. Thanks also for checking it on IDA 9.4. That 9.4 named the import itself is now noted on danielplohmann/smda#398.
Lint, the sync checks and the suite (334 tests) pass on this head, and it contains current main. The licensed IDA job, dispatched on df48778 from a branch of this repository, passed: the IDA 9.3 GUI test with 146 of 146 checks, plus the IDALib test (https://github.com/familiary/mcrit-plugin/actions/runs/38725755872).
This makes the IDA plugin as responsive as the Binary Ninja one: MCRIT requests no longer run on IDA's main thread, and Convert exports about twice as fast. It builds on the open request stack, whose fixes it needs; see "How to merge" below.
Commits
IdaBackend.run_requestran synchronously, so a slow server froze IDA, for about a minute when fetching labels for a large result. Requests now run on a pool of four threads, and a main-threadida_kernwin.register_timerhands each result toon_done. Results are delivered on the main thread, and no result is delivered while a dialog opened by another result is still open. Closing the form cancels queued requests and drops the results still to come.ida_domainbackend whenever that package is importable, and inside IDA its wrappers make the export about twice as slow. A subclass of smda's IDAPython interface reads instruction bytes with oneget_item_sizelookup and walks code references withida_xrefdirectly. The reports are byte-identical; this belongs upstream in smda eventually.use_smda_for_analysison, Convert exported the disassembler's analysis only to print how its function set differs from SMDA's. The newBackend.get_function_offsets()gives the same comparison in milliseconds; the export remains the fallback.run_backgroundnow runs its work on a worker on IDA too. The backend methods that read the database run on the main thread throughexecute_sync(MFF_READ). The export reads what smda's exporter asks for intoSmdaInterfaceSnapshotbehind a short wait box, then disassembles and hashes on the worker. The reports are byte-identical with smda 4.3.10 and 4.9.0; the main thread's share of the export is 26 to 35 percent.MatchTable.pyreplaces an item per cell, so 884 matches show in 0.23 s instead of 0.5 s._aggregates.Measurements
Export in IDA 9.4 with
ida_domaininstalled; the reports are byte-identical to smda's, checked on x86-64, arm64e and PE binaries:e0b6f853…(PE x86-64)Against mcrit.malpedia.io for sample
e0b6f853…(221,398 matches over 151,306 remote functions), with the stack and this PR every request below runs off IDA's main thread:The result download itself (11 MB, about 30 s) is bound by the server;
compact=Trueleaves out the function matches the Function Overview needs.In the IDA 9.4 GUI with sample
e0b6f853…against mcrit.malpedia.io, measured read-only (no upload, no job, no rename), longest stall of IDA's main thread per step, with a 10 ms timer:Before this branch, each of these requests held IDA for its whole duration. The remaining stalls of up to about a second are full garbage collections while large results load: the interpreter then tracks about a million objects, most of them SMDA and MCRIT records. Tuning or freezing the collector would change IDA's interpreter for every other plugin, so the branch leaves it alone.
How to merge
This branch contains the merges of #39, #38, #40, #34 and #36, which bring in #30, #31, #33, #35 and #37. Until they land, the diff here shows all of them.
main. The merge commits drop out and only the commits above remain; I'll push that rebase.Merging this PR first would merge the whole stack with it, unreviewed one by one, so please don't.
Review fixes from #30 and #31
The fixes for the reviews of #30 and #31 were merged up the stack and into this branch, with no force-push. One merge needed more than taking both sides. Block Scope now has two checks: commit 2's skip of a query whose function the cursor has left, and #31's request number, which drops answers for an older block size. They are combined into one
superseded()check. A query is skipped when it is picked up, its block lookups stop, and its answer is dropped as soon as either the function or the requested view has changed. On IDA, #31's other fixes also apply to this branch's worker pool:Testing
ruff format --check,ruff checkandpython -m pytest testspass at every commit (333 tests at the tip, after mergingmainwith Ask the server for a sample's jobs and download the family and sample lists after Convert #39, Tidy the match graph fetch: closing message, kept reason, no double opens #44 and Align the vendored McritClient with MCRIT 1.15.0's client #45).mainagainst MCRIT 1.15.0 with SMDA 4.10.0. The IDA API calls the branch adds were checked against the IDA 9.4 docstrings and python.docs.hex-rays.com. The GUI test now waits for each action's result instead of expecting it synchronously.