Repository navigation
Answer findLoops' dominators from the dominator tree, and bound it - #282
Conversation
cfg_explorer_detector.run() went straight to get_roots(), which falls back to min() of the graph's nodes when no node is a root. A graph with no nodes at all - an empty digraph, an empty body, or any text that is not a dot graph, all of which parse to an empty graph - made that min() raise ValueError, and /explore/findLoops/ answered with a 500 and a traceback in the log. A body that is not UTF-8 got the same 500 earlier, from the route's decode. The pages never post such a graph: both parse the fetched graph in the browser before posting it, and graphlibDot.parse() throws on an empty one, while a function without disassembly gets a one-node "No control flow graph stored" graph. So the 500 took a hand-made POST, which any logged-in visitor can send. A graph without blocks has no loops, so run() answers [] for it, the same answer a function without loops gets, and the route decodes the body with replacement, so bytes that are not UTF-8 parse to whatever blocks they have, or to none. main() carried a line-for-line copy of run()'s body; it now reads the file and calls run(), so it picks up the same guard.
With several roots get_roots() adds my_super_root above them. A posted graph with a block of that name made it one more root's parent instead, which left no root and failed the assert in run() with a 500. The super root now takes a name the graph does not use.
dominanators() kept one set per block and intersected the sets of its predecessors until nothing changed. On a posted graph that costs more than the square of its blocks: 26 s for 1000 blocks, 294 s for 2000, in one request from any visitor. The real functions of the test corpus, up to 971 blocks, took up to a second. The dominators now come from networkx's immediate_dominators, and membership is read off the dominator tree, with blocks the entry never reaches still dominated by every block as before. immediate_dominators is still quadratic on a graph built for it, and the loop bodies are the answer itself, which a graph where every block jumps back to the entry makes grow with the square of its blocks. So findLoops now refuses graphs over 8000 blocks or 16000 edges, and stops at 2000 back edges or 200000 loop body entries; such a graph gets no loops, so the page still draws it. addParentInfo stops recording "equal" entries, which the page does not read, past 200000, so a flattened dispatcher keeps its loops, and it looks headers up in sets instead of scanning node lists.
|
Thanks, the speed-up is real: on 20,000 random graphs (1-60 blocks, multi-root, rootless, partly unreachable) the
Merge position: before #292, since once large graphs draw without a click, every view of a big function POSTs to |
…parsing at the bounds The bounds were checked on the parsed graph, and MAX_CONTENT_LENGTH lets 1 GiB through: 32 MiB of edges built 1.6 GiB of graph before the block bound refused it. A body declaring more than FIND_LOOPS_MAX_BYTES (32 MiB) is now answered without being read, a chunked one is read only up to the cap, and the parser gives up as soon as the graph is past MAX_BLOCKS or MAX_EDGES. It scans for the lines that can hold a block or an edge rather than splitting the text first, which held 570 MiB for 32 MiB of short lines. A skipped analysis still answers 200 and [], now with an X-Loop-Analysis header naming the bound, for the page to read. The bounds test lowered MAX_BLOCKS for all four of its cases, so each of them hit the block bound; only the block case does now.
main.js and main_duo.js read findLoops' X-Loop-Analysis header and say above the graph that the function is too large for loop analysis, with its block count, and that its loops are not highlighted. The graph is drawn as before.
|
Both done. 89b7f01 caps the body.
00683ea shows the skip.
There are new tests for the header, the cap (asserting how far the stream was read) and the note in the browser. 15 fail before the change. Full suite: 2125 passed. On the merge order: #292 rewrites the |
This branch loads the function page's graph viewer, and so lays out and draws the graph, only once the graph comes into view. open_function() in the loop analysis note tests master gained with #282 waited for nodes that were never drawn; it now scrolls #xcfg_top into view after the goto, as this branch's own browser tests do.
The function comparison on this branch lays out its graphs, and loads its viewer, only once they come into view. open_comparison() in the loop analysis note tests master gained with #282 waited for nodes that were never drawn; it now scrolls the graphs into view after the goto, as testFunctionVsBrowser's scroll_to_the_graphs() does.
/explore/findLoops/takes a dot graph from any logged-in visitor and runs the loop detector on it. Itsdominanators()kept one set per block and intersected the sets of each block's predecessors until nothing changed, which costs more than the square of the block count. On a crafted chain with a back edge every ten blocks, master (1cdc195) took 26 s for 1000 blocks and 294 s for 2000 (a 30 KB body), in one request; 4000 blocks did not finish in 15 minutes. Real functions are cheaper but not cheap: on the live testbed (mcrit 1.13.0) the 13 functions of 500 or more blocks took 0.2 to 1.1 s each.Change.
dominanators()now answers from the dominator tree (mcritweb/views/cfg_explorer_detector.py:52-115): networkx'simmediate_dominatorsbuilds the tree, and a block dominates another when a depth-first walk of the tree enters it before and leaves it after the other. It keeps the interface its callers use,a in doms[b]. A block the entry never reaches was dominated by every block under the old sets, which never narrowed from "all blocks"; it still is. networkx before 3.6 lists the start block as its own immediate dominator and later versions leave it out; both are handled.immediate_dominatorsis itself quadratic on a graph built for it (a chain whose last block jumps to a leaf of every chain block: 1.8 s at 8000 blocks, 8.6 s at 16000), and the loop bodies are the answer itself, which a graph where every block jumps back to the entry makes grow with the square of its blocks (15.6 MB for 2000 blocks). So the request now passes these bounds (cfg_explorer_detector.py:37-45, checked at:302,:260,:269):flowchart LR A[posted dot graph] --> B{"over 8000 blocks<br/>or 16000 edges?"} B -- yes --> R["[] and a warning in the log"] B -- no --> C[dominator tree] C --> D{"over 2000<br/>back edges?"} D -- yes --> R D -- no --> E[loop bodies] E --> F{"over 200000<br/>body entries?"} F -- yes --> R F -- no --> G["parents and equal lists<br/>(equal cut at 200000)"] G --> H[loops as JSON]equalentries.[](mcritweb/views/explore.py:669-674). The page parses whatever findLoops answers (mcritweb/static/trace_CFG/main.js:1263), so a JSON answer is what keeps the graph drawn, without loop shading, instead of not at all. The page says nothing about the skipped loops; the log does.equallists are cut rather than refused. Loops of one length that share a header each list the later ones, so a flattened dispatcher with a few thousand such loops would make an answer the square of that. The page does not readequal(mcritweb/static/trace_CFG/loopCollapser.js:815is a TODO), so such a function keeps all its loops and loses only list entries past 200000.addParentInfolooks headers up in sets rather than scanning node lists; same answer.Verification. On the live testbed, master on :5000 against a side instance of this branch, same backend. For all 13 functions of 500 or more blocks, findLoops answered byte-identically on both, and the branch took 0.02 to 0.09 s where master took 0.2 to 1.1 s (function 9344, 971 blocks: 1.08 s against 0.056 s). The 700-block fan posted to the side instance answered
[]in 1.4 s and loggedfindLoops: graph of 15964 bytes not analysed: more than 200000 loop body entries. Only findLoops and fetchDotGraph were called; the corpus check afterwards reports it unchanged. The side instance ran 59116ba; the final commit adds the block and edge bounds and theequalcut, and gives the same answers for the three largest of those functions (checked offline against master's saved answers).Crafted graphs, run against the detector directly:
An independent review tried to break the equivalence on 3000 random graphs (unreachable cycles, edges from unreachable into reachable blocks, several or no roots) and against networkx 2.8.8, 3.4.2, 3.5, 3.6 and 3.7, and found no difference in any
a in doms[b]the callers ask.Tests. In
tests/testCfgLoopDetection.py:_reference_run, the oracle the equivalence tests compare against, now carries the old per-node dominator sets and the oldaddParentInfoas transcriptions, so the existing comparison over the five real fixture functions holds the new code to the old answers.test_loops_match_the_set_based_dominators_on_random_graphs(300 seeded graphs) andtest_dominator_membership_matches_the_set_based_dominators(every pair of blocks, 100 graphs) compare with the old algorithm directly. Both pass on Stop findLoops answering hand-made graphs with a 500 #270's code too, which is the point.test_a_graph_whose_answer_would_outgrow_the_bounds_gets_no_loopscovers the block, edge, body and back-edge bounds with a graph each (the block bound lowered to 80 in the test, since a graph over the real one keeps the old algorithm busy for hours), andtest_many_equal_loops_keep_their_loops_and_lose_only_equal_entriescovers theequalcut. Removing any one bound fails exactly its case.test_a_graph_just_inside_the_bounds_is_still_answeredchecks a 600-block fan still gets the old answer.Relation to open PRs. Stacked on #270 (
fix/findloops-empty-graph), which changesrun(),get_roots()and the findLoops view right next to these lines; merge #270 first. A trial merge (git merge-tree, against this head) with #268, #269 and #271 to #281 is clean.