Repository navigation
Fix label import, scope queries and settings that shipped wrong in 1.1.10 - #23
Conversation
4cb410d to
70c07ee
Compare
danielplohmann
left a comment
There was a problem hiding this comment.
Reviewed the three commits of this PR (pr/17..70c07ee). Nothing blocks the merge. There is one should-fix, and the question you asked is answered below. On the head, ruff passes, 214 pytest pass, and the settings and metadata sync scripts pass.
Should-fix
An active config_override.json now takes priority without telling anyone. See mcrit_plugin/core/config.py:105-118. Reading overrides first is the correct fix, but it changes who is affected:
docs/config_override.json.templatelists all 18 keys at their defaults, includingmcrit_server: 127.0.0.1and an empty token. The IDA ZIP ships it.- Someone who copied the whole template, or still has a full copy from 1.1.x (where it never applied), gets server, token and every option pinned after upgrading.
- From then on, Plugin Settings Manager, hcli and Binja Settings silently do nothing.
Suggested fix: print the forced keys once at load, e.g. [!] config_override.json forces: mcrit_server, .... Alternatively, cut the template down to one or two example keys. Doing both would be best.
Nits
config.py:110-116: if the JSON top level is not an object, the plugin fails at import. A number, bool or null makeskey in override_settingsraiseTypeError; a string does substring matching and then fails. OnlyJSONDecodeErrorandIOErrorare caught, so anisinstance(..., dict)check would fix it.BlockMatchWidget.py:244-255: not caching a failed PicBlockHash lookup is right. But if the server answers the family and sample lists and then times out on picblockhash, every revisit repeats every failed hash one after another on the UI thread. With live tracking on, that is N x timeout per function visit, where before only the first visit paid. Stopping after the first failure in a visit would fix it.FunctionMatchWidget.py:138-149and the same check inBlockMatchWidget: moving fromis Nonetonotis right for the "unknown hashes" fix. But a reachable server with 0 families or samples now shows "unavailable. Check server connection." and fetches both lists again on every query.BlockMatchWidget.py:171-179(low confidence):queryCurrentBlocktakes the block from the cursor, but the function comes only fromFunctionMatchWidget.hook_refresh. On Binja, where cursor notifications can arrive late, the pair can mismatch. Taking the function from the report entry that contains the block would rule that out.FunctionMatchWidget.py:205: the<= 10check is correct for default servers. We checked MCRIT'sMinHasher.isMinHashableFunction: it uses> MINHASH_FN_MIN_INS, or> MINHASH_FN_MIN_BLOCKS, which is off by default. It still copies a value the server can configure, so a comment there would help.- CHANGELOG: the Block Scope message fix ("Live Block Queries are deactivated") has no bullet. The #17 bullet "Where config_override.json applied, which was only when the settings store failed…" now stands next to the new precedence bullet, and the two read better as one.
- The sorted-table fix has only GUI-test coverage. An offscreen PySide6 test guarded by
importorskipis feasible: we built the realResultChooserDialogand a table with the realDropdownDelegateand_populatedRowin a scratch script, and it behaves as described.
Checked and correct
- Sorted tables, at runtime:
- After a descending sort,
getEditorForRow(_populatedRow(r))returns each row's own editor. Keyed by the visual row, every row got another row's editor. jobInfoAt, the status colour key and the selection all follow the job.- No other consumer indexes by the visual row.
- After a descending sort,
sample_group_only, checked in the MCRIT source: the query and match resources pass**getMatchingParamsintogetMatchesForSmdaFunction/getMatchesForSample, and neither accepts it. OnlygetMatchesCrossdoes. MCRITweb forwards neither request's parameters.- Override precedence: holds on all three backends. Boolean coercion handles
"false". - Runner refusal: limited to the four dev and CI runners, not the plugin runtime.
- Scope fixes: the cursor is tracked before Convert, and
[]versusNonecaching is correct. Sample Info falls back to the best family, and aNonejob list no longer opens the chooser. - QtShim: nothing references the removed attributes, and
idaVersionsis>=9.0.
Your question: sample_group_only
Remove it in 2.0.0.
- It is the major release, so removal is clean now; otherwise it waits for 3.0.
- The option broke every request against a plain MCRIT server, so no working setup depends on it.
- A toggle that does nothing, in both settings UIs, will produce "why does this do nothing" reports.
One check first: make sure hcli / ida-settings don't reject or loudly warn about an undeclared key still stored in an existing ida-config.json. If they error, keep it deprecated for 2.0.x after all. Either way, keep the sample_group_only parameters in the vendored McritClient, since the cross-compare path uses them.
70c07ee to
c900b2f
Compare
|
@r0ny123 #17 is merged into Please bring this branch up to date with
The licensed IDA job can't run from the fork, so I'll dispatch it again on an origin branch at the new head before merging, as for #17. #22 comes after this one. |
All of these behaved this way in 1.1.10. - Both the Function Overview and the result chooser allow sorting, which moves rows, while the Overview's label mappings and dropdown editors and the chooser's job_infos and status colours stayed keyed by the row each entry was filled into. After a header click, Import Labels applied each label to another function and the chooser loaded another job than the selected one. Each cell now carries the row it was filled into, or its job's index, and every lookup goes through that. Both GUI tests import labels from a table sorted by descending offset and check each renamed function against its own matches' labels; without the fix that check names three functions. - Function Scope's refresh returned before reading the cursor while no report existed, so Query Current Function and Query Current Block right after Convert found no current function. The cursor is followed first, and Query Current Block takes the block under the cursor. - MCRIT MinHashes a function only above 10 instructions and answers a query for one of exactly 10 with a 500; Function Scope now requires more than 10. - A failed PicBlockHash query was cached as None and read as "no matches"; only answers are cached now. - Function and Block Scope tested sample_infos against None, which it never is, so a failed sample list showed "unknown" hashes instead of "Remote family/sample info unavailable". - A failed job query opened the result chooser, which then claimed no matching results existed; the handler stops after the failure message. - Sample Info kept a family picked for an earlier result even when the next result has no match in it; it falls back to the best family. - Block Scope said "Live Function Queries are deactivated", and the IDA graph view's hover hint showed a placeholder; it shows the block's offset. SmdaGraphViewer's header no longer calls it a Hex-Rays example, and draw() no longer calls OnGetText only to discard it.
Neither did in 1.1.10. - MCRIT hands the query parameters of POST /query/function to MinHashIndex.getMatchesForSmdaFunction, and those of a single-sample matching job to Worker.getMatchesForSample. Neither takes sample_group_only, so with the setting on every Function Scope query answered 500 and every matching job failed. MCRIT's own client sends it only for cross compares, which the plugin does not request, and MCRITweb forwards neither request's parameters. The plugin no longer sends it; the setting stays declared, titled and documented as deprecated with no effect, and the vendored client's docstring says what MCRIT does with it. - config_override.json was merged into the defaults, which McritConfig reads only when the settings store raises. ida-settings answers with the declared default for any declared key, and Binary Ninja's Settings has every key once they are registered, so an override never applied. The overrides are kept apart and read first. README says the file's keys take precedence, so it should hold only the keys to force, and where it goes for Binary Ninja. - The config tests get an empty plugin root, so a developer's own config_override.json cannot change the defaults they pin. - An override in the plugin under test would now beat the settings the integration runners use, so all four refuse to start while one is there, before they install or write anything.
The plugin's first commit brought code over from IDAscope, written for IDA 6: attributes that kept json and os reachable on IDA 6.x for Windows, a datetime alias for the same reason, comments about shared MCRIT4IDA modules and IDA < 6.9, and in QtShim a PySide 1 branch for IDA 6.8 and below in each of the 50 accessors. The plugin requires IDA 9, nothing read the attributes, and the branches were unreachable. Every QtShim accessor still resolves under PySide6, and IDA 9.0/9.1 keep PyQt5. A comment typo is fixed on the way.
sample_group_only is removed. MCRIT fails every matching job and function query that carries it, so no working setup depends on it. With hcli 0.26 and ida-settings 3.5.1, a value that an earlier version stored stays in ida-config.json and is ignored: installing a version that no longer declares the key succeeds, and the remaining settings read, set and list as before. The vendored McritClient keeps its parameters, because MCRIT's cross compare takes them. config_override.json now takes precedence, so a full copy of the old template would pin every setting without anyone noticing. The plugin prints the keys an override forces when it loads. The template shows two keys instead of all eighteen, and an override whose top level is not a JSON object is ignored instead of failing the import. NumberQTableWidgetItem compared its text with float(), so sorting Function Scope's match table by its hex offsets raised inside Qt's sort, which crashes the process under PySide6 6.11. Hex text now compares by value. Scope queries: - An empty family or sample list from a reachable server counts as fetched; only a failed request (None) is fetched again. - A failed PicBlockHash lookup skips the rest of that visit's lookups instead of each waiting out the same timeout. - Query Current Block takes the function from the block under the cursor, since the Function Scope hook can lag behind on Binary Ninja. An offscreen PySide6 test covers the sorted Function Overview, the result chooser and hex sorting, and the pytest job installs PySide6 to run it. The IDA entry-point test hides Qt modules that other tests loaded, so it checks only the entry point's own imports.
c900b2f to
5b09e87
Compare
|
Thanks for the review. The branch is rebased onto
With MCRIT 1.12.0 and smda 4.9.0, the IDA 9.3 GUI test (145 checks), the IDALib test, the Binary Ninja GUI test (167 checks) and the headless test pass locally at |
danielplohmann
left a comment
There was a problem hiding this comment.
Approving at 5b09e87. It addresses the whole review: the forced override keys are printed at load, the template is minimal, a non-object override is ignored, sample_group_only is removed (and thanks for checking hcli / ida-settings with a stale key), and the scope nits and sorted-table test are in. The hex-sort crash fix is a good catch.
The licensed IDA 9.3 job ran on this exact commit from an origin branch against smda 4.9.0 (run 36474647820): 145 PASS, 0 FAIL.
One nit for later, not blocking: _numeric in NumberQTableWidgetItem.py still raises on non-numeric text ("", "unknown"), which would reach Qt's sort the same way. If any of those columns can ever hold such text, falling back to a text comparison would close it.
Based on
mainafter #17. It fixes behaviour that already shipped in 1.1.10 and turned up while testing #17, so it should merge beforeida-v2.0.0is tagged. #22 stacks on this one.The fixes users can notice have entries in the
[2.0.0]section ofCHANGELOG.md.What it fixes
job_infosand the status colours stayed keyed by the row each entry was filled into. After a header click, Import Labels applied each label to another function and the chooser loaded another job than the selected one. Each cell now carries the row it was filled into, or its job's index.sample_group_only. MCRIT hands the query parameters ofPOST /query/functiontoMinHashIndex.getMatchesForSmdaFunction, and those of a single-sample matching job toWorker.getMatchesForSample. Neither takessample_group_only, so with the setting on every Function Scope query answered 500 and every matching job failed. MCRIT's own client sends it only for cross compares, which the plugin does not request, and MCRITweb forwards neither request's parameters. The plugin no longer sends it, and the setting is removed (see "Review of the first three commits" below).config_override.jsonwas merged into the defaults, which are read only when the settings store raises, and ida-settings answers with the declared default for every declared key. So an override never applied, onmaineither. Overrides are now read first, and the plugin prints the keys an override forces when it loads. All four integration runners refuse to start while the plugin under test has one, and README says to keep only the keys to force.MINHASH_FN_MIN_INS, compared with>).Checks
Local at the head, against MCRIT 1.12.0 and SMDA 4.9.0: ruff clean, 210 pytest passed with PySide6 installed (207 and one skip without it), the IDA 9.3 GUI test in autonomous mode,
-A(145 checks) and the IDALib test pass, and the Binary Ninja 6.0 GUI test (167 checks) and the headless test pass. Both GUI tests now import labels from a table sorted by descending offset; without the sorted-table fix, that check names the three functions that got another function's label.Review of the first three commits
5b09e87addresses the review, on a branch rebased ontomain:config_override.jsonvisibility: the plugin prints[!] config_override.json forces: …once at load, and the template lists two example keys instead of all eighteen. An override whose top level is not a JSON object is ignored instead of failing the import.sample_group_onlyremoved. Checked with ida-hcli 0.26.0 and ida-settings 3.5.1 in a throwawayIDAUSR: after installing the current package with--config sample_group_only=true, uninstalling and installing a build that no longer declares the key succeeds. The stored value stays inida-config.jsonand is ignored, andhcli plugin configlist,getandsetwork for the remaining keys. Only a read of the removed key raisesKeyError, which nothing does now. hcli refusesplugin upgradefrom a local file, so the index upgrade path itself was not run. The vendoredMcritClientkeeps its parameters.config_override.jsonbullets are merged.NumberQTableWidgetItem, whose__lt__calledfloat(). Sorting that table by offset raised inside Qt's sort, which crashes the process under PySide6 6.11. That shipped in 1.1.10. Hex text now compares by value, with a test and a CHANGELOG bullet.