Skip to content

Reuse server connections and look up block hashes concurrently - #34

Merged
danielplohmann merged 4 commits into
familiary:mainfrom
r0ny123:mcrit-connection-reuse
Oct 12, 2026
Merged

danielplohmann merged 4 commits into
familiary:mainfrom
r0ny123:mcrit-connection-reuse

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

This builds on #31, which is its first commit; only 3a874a9 is new. It is independent of #33.

The vendored McritClient called requests.get and the other module-level helpers directly, so every request opened a new connection. Against an HTTPS server that means a TLS handshake per request, which against mcrit.malpedia.io cost most of a second each time. The client now keeps one requests.Session for its lifetime. Its interface is unchanged, and the core mcrit client could take the same change.

Block Scope pays that cost once per block. MCRIT has no batch query for block hashes, so Block Scope sends one request per distinct hash, and it sent them one after another. It now looks up the distinct uncached hashes on a pool of eight threads. As before, a failed lookup stays uncached so the next visit retries it, and it stops the lookups that have not started yet, so an unreachable server costs one timeout rather than one per block. On IDA, where run_request runs on the main thread, only the HTTP calls run on the pool.

sequenceDiagram
  participant B as Block Scope
  participant S as MCRIT server
  Note over B,S: before: one new TLS connection per hash, sequential
  B->>S: GET /query/picblockhash/h1
  S-->>B: matches
  B->>S: GET /query/picblockhash/h2
  S-->>B: matches
  Note over B,S: after: pooled keep-alive connections, eight lookups in flight
  par
    B->>S: h1
  and
    B->>S: h2
  and
    B->>S: … h8
  end
Loading

The table covers the block lookups for the largest function of sample e0b6f853… (3,955 instructions, 1,361 blocks) against mcrit.malpedia.io. They ran through the plugin's own _lookupBlockHashes:

blocks queried before keep-alive only keep-alive + 8 threads
166 distinct, blocks_min_size 4 (the default) 166 s 46 s 6.2 s
370 distinct, blocks_min_size 0 not run 11.8 s

The next table comes from a read-only Binary Ninja 6.0 GUI session against the same server and sample. That session loaded the sample's existing matching job and uploaded or created nothing. "Before" is #31, and "after" is this PR merged with #33, #35 and the SMDA change in danielplohmann/smda#378. The rows this PR accounts for:

step before: duration / longest UI stall after: duration / longest UI stall
Block Scope query, largest function 156.15 s / 0.23 s 5.92 s / 0.07 s
Function Scope query, largest function 3.30 s / 0.26 s 0.92 s / 0.04 s

The UI did not stall in either run, since #31 already moved these queries off the UI thread; what changes is how long the tables stay empty.

tests/ui_qt/test_block_lookups.py checks three things: each uncached hash is queried once, the lookups overlap, and a failure stays uncached and stops the lookups that have not started. tests/core/test_mcrit_client.py checks that the request helpers go through the client's session. All four fail against #31. pytest gives 214 passed, the Binary Ninja GUI test passes its 173 checks, and the settings and metadata sync checks and run_quality_checks.py pass.

The IDA GUI and IDALib tests were not run locally, because no licensed IDA was available. The client and the block lookup are shared with IDA, so please trigger the IDA workflow on this branch.

McritClient called requests.get and friends directly, so every request
opened a new connection, which against an HTTPS server means a new TLS
handshake. Against mcrit.malpedia.io that was most of a second per
request. The client now keeps one requests.Session for its lifetime.
Its interface is unchanged.

Block Scope sends one request per block hash, since MCRIT has no batch
query for them, and sent them one after another. It now looks up the
distinct uncached hashes on a pool of eight threads. As before, a
failed lookup stays uncached and stops the lookups that have not
started yet, so an unreachable server costs one timeout rather than one
per block. On IDA, where requests run on the main thread, only the
HTTP calls run on the pool.

Block lookups for the largest function (3,955 instructions, 1,361
blocks) of a 2,329-function sample, against mcrit.malpedia.io:

  blocks queried              before    after
  166 distinct (min size 4)   166 s     6.2 s
  370 distinct (min size 0)   not run   11.8 s

The sequential run for 370 hashes was not measured; at the observed rate
of one second per request it would take about six minutes.
# Conflicts:
#	CHANGELOG.md
danielplohmann
danielplohmann previously approved these changes Oct 12, 2026

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the numbers speak for themselves. I checked two risks of the shared requests.Session. Cookies: MCRITweb's API authenticates only by the apitoken header, never by its session cookie (mcritweb/__init__.py), so keeping cookies changes nothing. Threads: the pool of eight stays under urllib3's default of ten connections per host, so connections are reused rather than discarded. The new tests and the changed client test fail against main and pass with this PR.

Before merge, this needs main merged in. It conflicts in two files:

  • CHANGELOG.md: keep all the Unreleased/Changed bullets, main's first.
  • McritClient.__init__: keep main's raise_client_errors / raise_server_errors / self.timeout = timeout lines, and add your _session lines after them, dropping your self.timeout = None.

With that resolution, lint, the sync checks and the suite (229 tests) pass here. Please do the merge yourself: the ruleset only counts my approval when the last push isn't mine.

Not for this PR: the core mcrit client could take the same Session change, as you say. The interface is unchanged, so nothing has to move in lockstep.

@r0ny123
r0ny123 dismissed danielplohmann’s stale review October 12, 2026 07:56

The merge-base changed after approval.

The vendored client now creates its requests.Session next to the error
modes and timeout from the 1.12.0 alignment, and the request wrappers
from that alignment send through the session. The changelog keeps every
Unreleased/Changed bullet.
danielplohmann
danielplohmann previously approved these changes Oct 12, 2026

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The merge of main (d03d8fe) resolves both conflicts as suggested: every changelog bullet is kept, and _session comes after main's client attributes. Lint, the sync checks and the suite (229 tests) pass on it. Approving again.

@danielplohmann

Copy link
Copy Markdown
Collaborator

#35 is merged, so this needs main merged in once more. The only conflict is in CHANGELOG.md: keep every Unreleased/Changed bullet, main's first. I'll approve again after your push.

# Conflicts:
#	CHANGELOG.md

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The merge of main (6939992) keeps every changelog bullet, with main's first. Lint, the sync checks and the suite (229 tests) pass on it. Approving again.

@danielplohmann
danielplohmann merged commit 906fc0b into familiary:main Oct 12, 2026
10 checks passed
@r0ny123
r0ny123 deleted the mcrit-connection-reuse branch October 15, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants