Repository navigation
Reuse server connections and look up block hashes concurrently - #34
Conversation
McritClient called requests.get and friends directly, so every request opened a new connection, which against an HTTPS server means a new TLS handshake. Against mcrit.malpedia.io that was most of a second per request. The client now keeps one requests.Session for its lifetime. Its interface is unchanged. Block Scope sends one request per block hash, since MCRIT has no batch query for them, and sent them one after another. It now looks up the distinct uncached hashes on a pool of eight threads. As before, a failed lookup stays uncached and stops the lookups that have not started yet, so an unreachable server costs one timeout rather than one per block. On IDA, where requests run on the main thread, only the HTTP calls run on the pool. Block lookups for the largest function (3,955 instructions, 1,361 blocks) of a 2,329-function sample, against mcrit.malpedia.io: blocks queried before after 166 distinct (min size 4) 166 s 6.2 s 370 distinct (min size 0) not run 11.8 s The sequential run for 370 hashes was not measured; at the observed rate of one second per request it would take about six minutes.
# Conflicts: # CHANGELOG.md
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks, the numbers speak for themselves. I checked two risks of the shared requests.Session. Cookies: MCRITweb's API authenticates only by the apitoken header, never by its session cookie (mcritweb/__init__.py), so keeping cookies changes nothing. Threads: the pool of eight stays under urllib3's default of ten connections per host, so connections are reused rather than discarded. The new tests and the changed client test fail against main and pass with this PR.
Before merge, this needs main merged in. It conflicts in two files:
CHANGELOG.md: keep all the Unreleased/Changed bullets,main's first.McritClient.__init__: keepmain'sraise_client_errors/raise_server_errors/self.timeout = timeoutlines, and add your_sessionlines after them, dropping yourself.timeout = None.
With that resolution, lint, the sync checks and the suite (229 tests) pass here. Please do the merge yourself: the ruleset only counts my approval when the last push isn't mine.
Not for this PR: the core mcrit client could take the same Session change, as you say. The interface is unchanged, so nothing has to move in lockstep.
The merge-base changed after approval.
The vendored client now creates its requests.Session next to the error modes and timeout from the 1.12.0 alignment, and the request wrappers from that alignment send through the session. The changelog keeps every Unreleased/Changed bullet.
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks. The merge of main (d03d8fe) resolves both conflicts as suggested: every changelog bullet is kept, and _session comes after main's client attributes. Lint, the sync checks and the suite (229 tests) pass on it. Approving again.
|
#35 is merged, so this needs |
# Conflicts: # CHANGELOG.md
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks. The merge of main (6939992) keeps every changelog bullet, with main's first. Lint, the sync checks and the suite (229 tests) pass on it. Approving again.
This builds on #31, which is its first commit; only 3a874a9 is new. It is independent of #33.
The vendored
McritClientcalledrequests.getand the other module-level helpers directly, so every request opened a new connection. Against an HTTPS server that means a TLS handshake per request, which against mcrit.malpedia.io cost most of a second each time. The client now keeps onerequests.Sessionfor its lifetime. Its interface is unchanged, and the coremcritclient could take the same change.Block Scope pays that cost once per block. MCRIT has no batch query for block hashes, so Block Scope sends one request per distinct hash, and it sent them one after another. It now looks up the distinct uncached hashes on a pool of eight threads. As before, a failed lookup stays uncached so the next visit retries it, and it stops the lookups that have not started yet, so an unreachable server costs one timeout rather than one per block. On IDA, where
run_requestruns on the main thread, only the HTTP calls run on the pool.sequenceDiagram participant B as Block Scope participant S as MCRIT server Note over B,S: before: one new TLS connection per hash, sequential B->>S: GET /query/picblockhash/h1 S-->>B: matches B->>S: GET /query/picblockhash/h2 S-->>B: matches Note over B,S: after: pooled keep-alive connections, eight lookups in flight par B->>S: h1 and B->>S: h2 and B->>S: … h8 endThe table covers the block lookups for the largest function of sample
e0b6f853…(3,955 instructions, 1,361 blocks) against mcrit.malpedia.io. They ran through the plugin's own_lookupBlockHashes:blocks_min_size4 (the default)blocks_min_size0The next table comes from a read-only Binary Ninja 6.0 GUI session against the same server and sample. That session loaded the sample's existing matching job and uploaded or created nothing. "Before" is #31, and "after" is this PR merged with #33, #35 and the SMDA change in danielplohmann/smda#378. The rows this PR accounts for:
The UI did not stall in either run, since #31 already moved these queries off the UI thread; what changes is how long the tables stay empty.
tests/ui_qt/test_block_lookups.pychecks three things: each uncached hash is queried once, the lookups overlap, and a failure stays uncached and stops the lookups that have not started.tests/core/test_mcrit_client.pychecks that the request helpers go through the client's session. All four fail against #31.pytestgives 214 passed, the Binary Ninja GUI test passes its 173 checks, and the settings and metadata sync checks andrun_quality_checks.pypass.The IDA GUI and IDALib tests were not run locally, because no licensed IDA was available. The client and the block lookup are shared with IDA, so please trigger the IDA workflow on this branch.