Skip to content

Fetch labels in chunks and show each chunk as it arrives - #37

Merged
danielplohmann merged 6 commits into
familiary:mainfrom
r0ny123:binja-chunked-labels
Oct 12, 2026
Merged

danielplohmann merged 6 commits into
familiary:mainfrom
r0ny123:binja-chunked-labels

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

This builds on #35, which builds on #33 and #31; the diff shows their commits until they merge. Its own commits are 96cafe5 and ada2fef. The overview table model PR builds on this one. It was rebased onto #35 so that each label chunk's request carries the title #35 gives Backend.run_request; on #33 alone, merging #35 afterwards named the request with a variable this PR removes, and the fetch raised NameError.

flowchart LR
  main --> 31["#31 requests off the UI thread"] --> 33["#33 label fetch off the UI thread"] --> 35["#35 status bar tasks"] --> this["this PR: chunked label fetch"] --> next["table model for the Function Overview"]
Loading

#33 moved "Fetch labels for matches" off the UI thread, but it is still one request. For matching job 11353 of sample e0b6f853… on mcrit.malpedia.io, that request covers 151,306 matched functions, and no label shows for about 60 s. The fetch now requests the best-scoring ids first, in chunks: 2,000 in the first request, then 50,000 per request. The overview shows each chunk's labels as it arrives, at most every 2 s and always after the last chunk. A failed chunk stops the fetch and re-enables the button. The ids already answered count as fetched, so the next click asks only for the rest; a click after a complete fetch asks for all of them again, to pick up new labels.

The chunk sizes follow the server's cost. Timed against the server, a label request costs about 0.7 s plus 0.45 ms per id, and parallel requests do not help because the server handles them one after another. Small chunks therefore show the first labels early but add the fixed cost many times. Fetching all 151,306 ids in Binary Ninja 6.0 against mcrit.malpedia.io:

chunks first labels all labels requests
one request (#33) 60 s 60 s 1
2,000 each (server-timed) 1.8 s about 137 s 76
2,000, then 15,000 3.7 s 110 s 11
2,000, then 50,000 (this PR) 2.6–7.2 s 71–92 s 4

The ranges are from separate GUI runs on different days; the server's response times varied by up to a factor of two between them.

Each render rebuilds the whole overview. On this PR alone, that is still the QTableWidget with a combo box per row, and the longest UI stall while labels arrived was 1.1 to 5.6 s across runs. The table model PR on top of this one brings that down to 0.12 s.

tests/ui_qt/test_function_overview.py covers these cases:

  • chunks go out best match first, with the button disabled until the last one answers;
  • the later chunks are larger than the first;
  • chunks between two renders do not repopulate the table;
  • a chunk that succeeds and a next one that fails: the answered ids stay marked, the button is enabled, and the next click asks only for the missing ids;
  • a click after a complete fetch asks for every id again;
  • the next chunk is requested before the current one renders, a new fetch shows its first chunk at once, and the activity line says how the fetch ended.

The Binary Ninja GUI test passes all 176 checks against a local MCRIT 1.12.0. After merging main, pytest gives 234 passed, and run_quality_checks.py and the settings and metadata sync checks pass. The IDA tests were not run. On IDA the chunks run synchronously one after another, as the single request did.

The label fetch asked for every matched function in one request, which
took about a minute on a result with 150,000 matches before any label
showed. It now requests 2,000 ids at a time, best matches first, and
renders the overview when a chunk arrives, at most every two seconds and
always after the last. A failed chunk stops the fetch and leaves the
rest to be requested again.
Each label request costs the server a fixed amount on top of the ids
it resolves, and parallel requests do not help, since the server
handles them one after another. Fixed chunks of 2,000 ids put the first
labels at about 2 s but raised the total for 151,306 ids from 60 s to
about 137 s. The first request stays at 2,000 ids and the later ones
take 50,000.

Fetching the labels of matching job 11353 on mcrit.malpedia.io in
Binary Ninja 6.0:

  chunks                    first labels   total   table renders
  one request (before)      60 s           60 s    1
  2,000 then 15,000         3.7 s          110 s   11
  2,000 then 50,000         3.1 s          78 s    4
@r0ny123
r0ny123 force-pushed the binja-chunked-labels branch from 27c31b5 to ada2fef Compare October 1, 2026 19:46

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, chunking the fetch makes sense, and the numbers in the description help. It merges into main without conflicts. Lint, the sync checks and the suite (230 tests) pass on the merged tree. One thing before merge.

The next click does not ask only for the rest. The button calls fetchLabels(force=True) (FunctionOverviewWidget.py:225), and force=True clears _label_requested_ids, so a click after a failed chunk asks for everything again. Only the automatic fetch (force=False) skips the ids already answered. test_a_failed_chunk_stops_the_fetch_and_is_retried_later also passes against main's logic: its first chunk fails, so nothing is ever marked, and the force reset empties the set anyway. Please let one chunk succeed and the next fail, then check that the answered ids stay marked, the button is enabled, and fetchLabels() asks only for the rest. The description and the changelog sentence ("requested again by the next click") need the same correction.

Not blocking, since #41 makes IDA's run_request asynchronous: until then IDA runs the chunks one after another on the main thread. Each 50,000-id chunk takes longer than LABEL_RENDER_INTERVAL, so the table is rebuilt after every chunk without a repaint in between, and the freeze gets longer than the single request was. If #41 is held back for any reason, IDA should fetch in one request, and the changelog bullet should start with "Binary Ninja:".

Smaller points:

  • Start the next chunk's request before _showFetchedLabels(), so the server works while the table renders.
  • Reset _label_last_render when a fetch starts. Otherwise a fetch that starts within 2 s of the previous render does not show its first chunk at once.
  • The activity line stays at "Fetching labels: X of N..." after the last chunk or a failure. A closing line ("Fetched labels for N functions", or where it stopped) would help.
  • The generation counter fixes the early re-enable when an automatic fetch overlaps a running one, but the stale chunk's request is still sent and the server answers it first. A comment saying so would help; there is nothing to fix, since a request cannot be cancelled.

The button calls fetchLabels(force=True), and force cleared the ids
already answered, so after a failed chunk the next click asked for every
matched function again instead of the rest. force now clears them only
when the last fetch completed: after a stop the click continues where
the fetch stopped, and after a complete fetch it refreshes all labels.

The fetch also:
- requests the next chunk before rendering the current one, so the
  server works while the table renders;
- resets the render clock when it starts, so its first chunk shows at
  once even right after another render;
- ends the activity line with how it ended, "Fetched labels for N
  matched functions." or where it stopped.

The old failed-chunk test passed against the previous logic too, since
its first chunk failed and nothing was ever marked. The new one lets a
chunk succeed and the next fail, then checks that the answered ids stay
marked, the button is enabled and the click asks only for the rest.
@r0ny123

r0ny123 commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. I merged main in, and bfb0dab addresses the review.

  • The next click asks only for the rest. force now clears the answered ids only when the last fetch completed. After a failed chunk, the click continues where the fetch stopped. After a complete fetch, it asks for everything again, so new labels on the server still show up. The new test lets one chunk succeed and the next fail. It checks that the answered ids stay marked, the button is enabled, and the next click asks only for the rest. It fails against the old logic. The changelog and the description now say the same.
  • The next chunk is requested before the current one renders.
  • _label_last_render is reset when a fetch starts, so a new fetch shows its first chunk at once.
  • The activity line ends with "Fetched labels for N matched functions." or with where the fetch stopped.
  • A comment at the generation check notes that the stale chunk's request is sent anyway, since a request cannot be cancelled.

On IDA: #41 makes IDA's run_request asynchronous and is meant to follow the stack, so I left the chunking as it is. If #41 is held back, I'll switch IDA to a single request and start the changelog bullet with "Binary Ninja:".

Lint, the sync checks and the suite (234 tests) pass on the merged tree.

# Conflicts:
#	CHANGELOG.md

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that resolves it. The stopped flag makes a click continue where a failed fetch stopped, and a click after a complete fetch still refreshes everything. Requesting the next chunk before rendering also helps IDA until #41: with the synchronous run_request the chain now runs to the last chunk first, and only that last chunk renders. The four new tests fail against the previous logic. Lint, the sync checks and the suite (237 tests) pass on the branch, which merges into main without conflicts.

@danielplohmann
danielplohmann merged commit 173b1e4 into familiary:main Oct 12, 2026
10 checks passed
@r0ny123
r0ny123 deleted the binja-chunked-labels branch October 15, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants