Repository navigation
Fetch labels in chunks and show each chunk as it arrives - #37
Conversation
The label fetch asked for every matched function in one request, which took about a minute on a result with 150,000 matches before any label showed. It now requests 2,000 ids at a time, best matches first, and renders the overview when a chunk arrives, at most every two seconds and always after the last. A failed chunk stops the fetch and leaves the rest to be requested again.
Each label request costs the server a fixed amount on top of the ids it resolves, and parallel requests do not help, since the server handles them one after another. Fixed chunks of 2,000 ids put the first labels at about 2 s but raised the total for 151,306 ids from 60 s to about 137 s. The first request stays at 2,000 ids and the later ones take 50,000. Fetching the labels of matching job 11353 on mcrit.malpedia.io in Binary Ninja 6.0: chunks first labels total table renders one request (before) 60 s 60 s 1 2,000 then 15,000 3.7 s 110 s 11 2,000 then 50,000 3.1 s 78 s 4
27c31b5 to
ada2fef
Compare
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks, chunking the fetch makes sense, and the numbers in the description help. It merges into main without conflicts. Lint, the sync checks and the suite (230 tests) pass on the merged tree. One thing before merge.
The next click does not ask only for the rest. The button calls fetchLabels(force=True) (FunctionOverviewWidget.py:225), and force=True clears _label_requested_ids, so a click after a failed chunk asks for everything again. Only the automatic fetch (force=False) skips the ids already answered. test_a_failed_chunk_stops_the_fetch_and_is_retried_later also passes against main's logic: its first chunk fails, so nothing is ever marked, and the force reset empties the set anyway. Please let one chunk succeed and the next fail, then check that the answered ids stay marked, the button is enabled, and fetchLabels() asks only for the rest. The description and the changelog sentence ("requested again by the next click") need the same correction.
Not blocking, since #41 makes IDA's run_request asynchronous: until then IDA runs the chunks one after another on the main thread. Each 50,000-id chunk takes longer than LABEL_RENDER_INTERVAL, so the table is rebuilt after every chunk without a repaint in between, and the freeze gets longer than the single request was. If #41 is held back for any reason, IDA should fetch in one request, and the changelog bullet should start with "Binary Ninja:".
Smaller points:
- Start the next chunk's request before
_showFetchedLabels(), so the server works while the table renders. - Reset
_label_last_renderwhen a fetch starts. Otherwise a fetch that starts within 2 s of the previous render does not show its first chunk at once. - The activity line stays at "Fetching labels: X of N..." after the last chunk or a failure. A closing line ("Fetched labels for N functions", or where it stopped) would help.
- The generation counter fixes the early re-enable when an automatic fetch overlaps a running one, but the stale chunk's request is still sent and the server answers it first. A comment saying so would help; there is nothing to fix, since a request cannot be cancelled.
The button calls fetchLabels(force=True), and force cleared the ids already answered, so after a failed chunk the next click asked for every matched function again instead of the rest. force now clears them only when the last fetch completed: after a stop the click continues where the fetch stopped, and after a complete fetch it refreshes all labels. The fetch also: - requests the next chunk before rendering the current one, so the server works while the table renders; - resets the render clock when it starts, so its first chunk shows at once even right after another render; - ends the activity line with how it ended, "Fetched labels for N matched functions." or where it stopped. The old failed-chunk test passed against the previous logic too, since its first chunk failed and nothing was ever marked. The new one lets a chunk succeed and the next fail, then checks that the answered ids stay marked, the button is enabled and the click asks only for the rest.
|
Thanks. I merged
On IDA: #41 makes IDA's Lint, the sync checks and the suite (234 tests) pass on the merged tree. |
# Conflicts: # CHANGELOG.md
danielplohmann
left a comment
There was a problem hiding this comment.
Thanks, that resolves it. The stopped flag makes a click continue where a failed fetch stopped, and a click after a complete fetch still refreshes everything. Requesting the next chunk before rendering also helps IDA until #41: with the synchronous run_request the chain now runs to the last chunk first, and only that last chunk renders. The four new tests fail against the previous logic. Lint, the sync checks and the suite (237 tests) pass on the branch, which merges into main without conflicts.
This builds on #35, which builds on #33 and #31; the diff shows their commits until they merge. Its own commits are 96cafe5 and ada2fef. The overview table model PR builds on this one. It was rebased onto #35 so that each label chunk's request carries the title #35 gives
Backend.run_request; on #33 alone, merging #35 afterwards named the request with a variable this PR removes, and the fetch raisedNameError.#33 moved "Fetch labels for matches" off the UI thread, but it is still one request. For matching job 11353 of sample
e0b6f853…on mcrit.malpedia.io, that request covers 151,306 matched functions, and no label shows for about 60 s. The fetch now requests the best-scoring ids first, in chunks: 2,000 in the first request, then 50,000 per request. The overview shows each chunk's labels as it arrives, at most every 2 s and always after the last chunk. A failed chunk stops the fetch and re-enables the button. The ids already answered count as fetched, so the next click asks only for the rest; a click after a complete fetch asks for all of them again, to pick up new labels.The chunk sizes follow the server's cost. Timed against the server, a label request costs about 0.7 s plus 0.45 ms per id, and parallel requests do not help because the server handles them one after another. Small chunks therefore show the first labels early but add the fixed cost many times. Fetching all 151,306 ids in Binary Ninja 6.0 against mcrit.malpedia.io:
The ranges are from separate GUI runs on different days; the server's response times varied by up to a factor of two between them.
Each render rebuilds the whole overview. On this PR alone, that is still the
QTableWidgetwith a combo box per row, and the longest UI stall while labels arrived was 1.1 to 5.6 s across runs. The table model PR on top of this one brings that down to 0.12 s.tests/ui_qt/test_function_overview.pycovers these cases:The Binary Ninja GUI test passes all 176 checks against a local MCRIT 1.12.0. After merging
main,pytestgives 234 passed, andrun_quality_checks.pyand the settings and metadata sync checks pass. The IDA tests were not run. On IDA the chunks run synchronously one after another, as the single request did.