Skip to content

Answer findLoops' dominators from the dominator tree, and bound it - #282

Merged
danielplohmann merged 7 commits into
masterfrom
fix/findloops-dominator-tree
Oct 29, 2026
Merged

danielplohmann merged 7 commits into
masterfrom
fix/findloops-dominator-tree

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

/explore/findLoops/ takes a dot graph from any logged-in visitor and runs the loop detector on it. Its dominanators() kept one set per block and intersected the sets of each block's predecessors until nothing changed, which costs more than the square of the block count. On a crafted chain with a back edge every ten blocks, master (1cdc195) took 26 s for 1000 blocks and 294 s for 2000 (a 30 KB body), in one request; 4000 blocks did not finish in 15 minutes. Real functions are cheaper but not cheap: on the live testbed (mcrit 1.13.0) the 13 functions of 500 or more blocks took 0.2 to 1.1 s each.

Change.

  • dominanators() now answers from the dominator tree (mcritweb/views/cfg_explorer_detector.py:52-115): networkx's immediate_dominators builds the tree, and a block dominates another when a depth-first walk of the tree enters it before and leaves it after the other. It keeps the interface its callers use, a in doms[b]. A block the entry never reaches was dominated by every block under the old sets, which never narrowed from "all blocks"; it still is. networkx before 3.6 lists the start block as its own immediate dominator and later versions leave it out; both are handled.
  • immediate_dominators is itself quadratic on a graph built for it (a chain whose last block jumps to a leaf of every chain block: 1.8 s at 8000 blocks, 8.6 s at 16000), and the loop bodies are the answer itself, which a graph where every block jumps back to the entry makes grow with the square of its blocks (15.6 MB for 2000 blocks). So the request now passes these bounds (cfg_explorer_detector.py:37-45, checked at :302, :260, :269):
flowchart LR
    A[posted dot graph] --> B{"over 8000 blocks<br/>or 16000 edges?"}
    B -- yes --> R["[] and a warning in the log"]
    B -- no --> C[dominator tree]
    C --> D{"over 2000<br/>back edges?"}
    D -- yes --> R
    D -- no --> E[loop bodies]
    E --> F{"over 200000<br/>body entries?"}
    F -- yes --> R
    F -- no --> G["parents and equal lists<br/>(equal cut at 200000)"]
    G --> H[loops as JSON]
Loading
  • The largest real functions are far inside them: 971 blocks, 1539 edges, 86 loops, 3340 body entries, at most 5 equal entries.
  • Over a bound, the route logs a warning and answers [] (mcritweb/views/explore.py:669-674). The page parses whatever findLoops answers (mcritweb/static/trace_CFG/main.js:1263), so a JSON answer is what keeps the graph drawn, without loop shading, instead of not at all. The page says nothing about the skipped loops; the log does.
  • equal lists are cut rather than refused. Loops of one length that share a header each list the later ones, so a flattened dispatcher with a few thousand such loops would make an answer the square of that. The page does not read equal (mcritweb/static/trace_CFG/loopCollapser.js:815 is a TODO), so such a function keeps all its loops and loses only list entries past 200000.
  • addParentInfo looks headers up in sets rather than scanning node lists; same answer.

Verification. On the live testbed, master on :5000 against a side instance of this branch, same backend. For all 13 functions of 500 or more blocks, findLoops answered byte-identically on both, and the branch took 0.02 to 0.09 s where master took 0.2 to 1.1 s (function 9344, 971 blocks: 1.08 s against 0.056 s). The 700-block fan posted to the side instance answered [] in 1.4 s and logged findLoops: graph of 15964 bytes not analysed: more than 200000 loop body entries. Only findLoops and fetchDotGraph were called; the corpus check afterwards reports it unchanged. The side instance ran 59116ba; the final commit adds the block and edge bounds and the equal cut, and gives the same answers for the three largest of those functions (checked offline against master's saved answers).

Crafted graphs, run against the detector directly:

graph master (1cdc195) this branch
chain, 1000 blocks 26.3 s 0.06 s
chain, 2000 blocks 293.7 s 0.08 s
fan (every block back to the entry), 600 blocks not run 1.1 s, 1.3 MB answer
fan, 700 blocks not run refused after 1.2 s
star (one header, equal loops), 1999 loops not run 0.8 s, all loops, 1.2 MB answer
leaves (dominator worst case), 7998 / 8002 blocks not run 1.8 s / refused at once
nested headers over a shared 230-block body not run 2.8 s, 1.5 MB answer

An independent review tried to break the equivalence on 3000 random graphs (unreachable cycles, edges from unreachable into reachable blocks, several or no roots) and against networkx 2.8.8, 3.4.2, 3.5, 3.6 and 3.7, and found no difference in any a in doms[b] the callers ask.

Tests. In tests/testCfgLoopDetection.py:

  • _reference_run, the oracle the equivalence tests compare against, now carries the old per-node dominator sets and the old addParentInfo as transcriptions, so the existing comparison over the five real fixture functions holds the new code to the old answers.
  • test_loops_match_the_set_based_dominators_on_random_graphs (300 seeded graphs) and test_dominator_membership_matches_the_set_based_dominators (every pair of blocks, 100 graphs) compare with the old algorithm directly. Both pass on Stop findLoops answering hand-made graphs with a 500 #270's code too, which is the point.
  • test_a_graph_whose_answer_would_outgrow_the_bounds_gets_no_loops covers the block, edge, body and back-edge bounds with a graph each (the block bound lowered to 80 in the test, since a graph over the real one keeps the old algorithm busy for hours), and test_many_equal_loops_keep_their_loops_and_lose_only_equal_entries covers the equal cut. Removing any one bound fails exactly its case. test_a_graph_just_inside_the_bounds_is_still_answered checks a 600-block fan still gets the old answer.

Relation to open PRs. Stacked on #270 (fix/findloops-empty-graph), which changes run(), get_roots() and the findLoops view right next to these lines; merge #270 first. A trial merge (git merge-tree, against this head) with #268, #269 and #271 to #281 is clean.

cfg_explorer_detector.run() went straight to get_roots(), which falls
back to min() of the graph's nodes when no node is a root. A graph with
no nodes at all - an empty digraph, an empty body, or any text that is
not a dot graph, all of which parse to an empty graph - made that min()
raise ValueError, and /explore/findLoops/ answered with a 500 and a
traceback in the log. A body that is not UTF-8 got the same 500 earlier,
from the route's decode.

The pages never post such a graph: both parse the fetched graph in the
browser before posting it, and graphlibDot.parse() throws on an empty
one, while a function without disassembly gets a one-node "No control
flow graph stored" graph. So the 500 took a hand-made POST, which any
logged-in visitor can send.

A graph without blocks has no loops, so run() answers [] for it, the
same answer a function without loops gets, and the route decodes the
body with replacement, so bytes that are not UTF-8 parse to whatever
blocks they have, or to none. main() carried a line-for-line copy of
run()'s body; it now reads the file and calls run(), so it picks up the
same guard.
With several roots get_roots() adds my_super_root above them. A posted
graph with a block of that name made it one more root's parent instead,
which left no root and failed the assert in run() with a 500. The super
root now takes a name the graph does not use.
dominanators() kept one set per block and intersected the sets of its
predecessors until nothing changed. On a posted graph that costs more
than the square of its blocks: 26 s for 1000 blocks, 294 s for 2000, in
one request from any visitor. The real functions of the test corpus,
up to 971 blocks, took up to a second. The dominators now come from
networkx's immediate_dominators, and membership is read off the
dominator tree, with blocks the entry never reaches still dominated by
every block as before.

immediate_dominators is still quadratic on a graph built for it, and
the loop bodies are the answer itself, which a graph where every block
jumps back to the entry makes grow with the square of its blocks. So
findLoops now refuses graphs over 8000 blocks or 16000 edges, and stops
at 2000 back edges or 200000 loop body entries; such a graph gets no
loops, so the page still draws it. addParentInfo stops recording
"equal" entries, which the page does not read, past 200000, so a
flattened dispatcher keeps its loops, and it looks headers up in sets
instead of scanning node lists.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Thanks, the speed-up is real: on 20,000 random graphs (1-60 blocks, multi-root, rootless, partly unreachable) the
output is byte-identical to master's, and a 1,489-block CFG goes from 3.2 s to 0.02 s. Two things before it goes in:

  1. Cap the body before parsing. The size bounds are checked only after the whole POST body has been parsed into a
    networkx graph. The route is @visitor_required and MAX_CONTENT_LENGTH is 1 GiB. Measured with run() on a
    chain body: 18 MiB costs 4.5 s and +915 MiB RSS before "rejected: 1000001 blocks", and 78 MiB costs 20.8 s and
    +3.7 GiB. A logged-in visitor looping large POSTs to /explore/findLoops/ pins a gunicorn worker for minutes and
    can OOM it. Master is just as exposed, but this is the PR that bounds the route. Please answer before reading
    request.data when request.content_length exceeds a small cap (a real 8,000-block dot is a few MB with
    instruction labels; 32 MiB is generous), or count lines/edges inside parse_dot_to_graph.
  2. Say when loop analysis was skipped. When a bound hits, the route answers 200 [], logs a warning, and the
    graph draws without loop highlighting. Nothing on the page says why. I'd like a visible note, e.g.
    "This function is too large for loop analysis (N blocks); loops are not highlighted." Keep the 200 so the page
    still draws, but tell the page, e.g. with a field or header the JS can read.

Merge position: before #292, since once large graphs draw without a click, every view of a big function POSTs to
findLoops.

…parsing at the bounds

The bounds were checked on the parsed graph, and MAX_CONTENT_LENGTH lets 1 GiB
through: 32 MiB of edges built 1.6 GiB of graph before the block bound refused
it. A body declaring more than FIND_LOOPS_MAX_BYTES (32 MiB) is now answered
without being read, a chunked one is read only up to the cap, and the parser
gives up as soon as the graph is past MAX_BLOCKS or MAX_EDGES. It scans for the
lines that can hold a block or an edge rather than splitting the text first,
which held 570 MiB for 32 MiB of short lines.

A skipped analysis still answers 200 and [], now with an X-Loop-Analysis header
naming the bound, for the page to read. The bounds test lowered MAX_BLOCKS for
all four of its cases, so each of them hit the block bound; only the block case
does now.
main.js and main_duo.js read findLoops' X-Loop-Analysis header and say above
the graph that the function is too large for loop analysis, with its block
count, and that its loops are not highlighted. The graph is drawn as before.
@r0ny123

r0ny123 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Both done.

89b7f01 caps the body.

  • A Content-Length over FIND_LOOPS_MAX_BYTES (32 MiB, in create_app's defaults) is answered without the body being read, and a body without a length is read to at most the cap plus one byte.
  • The cap alone still let a body just under 32 MiB cost +1.6 GiB, so parse_dot_to_graph also stops as soon as the graph passes MAX_BLOCKS or MAX_EDGES, and it only parses lines with -> or [shape=.
  • 32 MiB of edges went from 49 s and +1.6 GiB to 0.08 s with no RSS growth, and 32 MiB of junk lines from 11.7 s and +567 MiB to 1.3 s. That was on a loaded machine, so the absolute times run high.

00683ea shows the skip.

  • A skipped analysis still answers 200 and [], now with X-Loop-Analysis: skipped; max-blocks=8000 (or whichever bound was hit).
  • The function page and both panels of the comparison read that header and say "This function is too large for loop analysis (N blocks); loops are not highlighted."
  • The bounds test used to lower MAX_BLOCKS for all four of its cases, so each case hit the block bound. Now each case hits its own bound.

There are new tests for the header, the cap (asserting how far the stream was read) and the note in the browser. 15 fail before the change. Full suite: 2125 passed.

On the merge order: #292 rewrites the main.js callback that holds the one-line call. When #292 takes master after this PR, the resolution is #292's side plus noteSkippedLoopAnalysis(result); after loopsObj = JSON.parse(...) in findLoopsAndShowGraph. I tried that merge, and the 102 tests it touches pass, #292's browser tests included.

@danielplohmann
danielplohmann merged commit 7ecbdaa into master Oct 29, 2026
8 checks passed
danielplohmann added a commit that referenced this pull request Oct 29, 2026
r0ny123 added a commit that referenced this pull request Oct 30, 2026
This branch loads the function page's graph viewer, and so lays out and
draws the graph, only once the graph comes into view. open_function() in
the loop analysis note tests master gained with #282 waited for nodes
that were never drawn; it now scrolls #xcfg_top into view after the
goto, as this branch's own browser tests do.
r0ny123 added a commit that referenced this pull request Oct 30, 2026
The function comparison on this branch lays out its graphs, and loads
its viewer, only once they come into view. open_comparison() in the loop
analysis note tests master gained with #282 waited for nodes that were
never drawn; it now scrolls the graphs into view after the goto, as
testFunctionVsBrowser's scroll_to_the_graphs() does.
@r0ny123
r0ny123 deleted the fix/findloops-dominator-tree branch October 30, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants