Performance benchmark suite + O(1) wildcard resolution + faster layout crawl - #3954
Open
T4rk1n wants to merge 7 commits into
Open
Performance benchmark suite + O(1) wildcard resolution + faster layout crawl#3954T4rk1n wants to merge 7 commits into
T4rk1n wants to merge 7 commits into
Conversation
Dash performance benchmarks
growth = late-third / early-third per-op time; ~1 is flat, a large value means the per-op cost scales with accumulated state. machine scale vs baseline: 2.10x - divided out of the baseline ratios so they compare like for like (the absolute warn/fail ceilings are left un-scaled); calibrated on |
A standalone benchmark harness for the renderer's hot paths, kept out of the pytest suite on purpose: timing is noisy, so it reports rather than flaking the test matrix. - benchmarks/scenarios.py: 10 scenarios across the platform - initial/deep hydration, Patch append (top-level + nested), scalar Patch update, full-list replacement (contrast), callback fan-out, ALL-wildcard resolution, and a deep callback chain. Each is a real Dash app + a browser-side interaction with warn/fail thresholds. - benchmarks/run.py: runs each scenario in its own bench_app subprocess against the production bundle, driven by headless Chrome. Timings use in-page performance.now() (no selenium-poll latency), aggregated as median/p90/max plus a growth ratio (late vs early per-op time) that flags O(total) creep. Also gates against a committed baseline and CPU-profiles a scenario (--profile) into a .cpuprofile + a hottest-functions table. - benchmarks/baseline.json: reference numbers on ubuntu-latest-class hardware. - .github/workflows/benchmarks.yml: PR job that builds the production renderer, runs the harness vs the baseline, hard-fails only on an order-of-magnitude regression, warns (without failing) on smaller drift, and upserts a sticky PR comment with the results table. - .ai/PERFORMANCE.md: how to run, how to profile, and the findings - including that ALL/MATCH wildcard resolution is O(n^2) (linear deep-equals getPath over the objs table; a hash index would make it O(1)), that post-fix Patch append has no single hotspot left, and that layouts deeper than ~250 fail to serialize. Removes the earlier pytest timing guards (tests/integration/renderer/ test_patch_append_perf.py); that coverage now lives in the harness + CI job.
Profiling the wildcard benchmark (one input change resolving an ALL callback
over 400 components) showed ~45% of the time in ramda `_equals`/`_functionName`:
getPath for a pattern-matching (dict) id did a linear `find(propEq(values,
'values'), keyPaths)` over every component sharing that id's key set, and it is
called once per resolved component - so an ALL dispatch over N components was
O(N^2) deep-equality comparisons.
The paths table now carries `objIndex`: {[keyStr]: {[valuesKey]: path}} where
valuesKey is JSON.stringify(values), so getPath is an O(1) map lookup. The
ordered `objs` array is untouched (resolveDeps / getAllPMCIds still walk it in
order for MATCH/ALLSMALLER); objIndex is only for exact lookups. It is
maintained inline in computePaths - copy-on-write per keyStr, so re-resolving
one chunk doesn't rebuild the index for unrelated components - and extended
incrementally in appendPaths. A table without an index (initial empty state,
hand-built test fixture) makes getPath fall back to the linear scan, so the two
can never disagree.
Result on the benchmark: wildcard_all_resolve ~580ms -> ~140ms (~4x, isolated),
with the _equals/_functionName frames gone from the profile. Renderer unit
suite 55/55; integration green across test_wildcards, multiple_callbacks,
layout_paths_with_callbacks, basic_callback, patch, children_reorder (69).
Baseline regenerated (full-suite numbers run hotter than isolated, but the
wildcard drop from 632ms to 217ms p90 is clearly captured); loosened the
intentionally-slow full_children_replace reference threshold to match.
Profiling every benchmark showed ~10-15% in ramda's curry machinery
(f1/f2/f3, _isPlaceholder, curried path/pathOr). It came from crawlLayout -
run on every component on every path recompute and callback gather - calling
curried path(['props','children'], obj)/pathOr(...) per node, plus the
path(['props','id'], child) in each crawl callback (paths.js, dependencies.js).
Replace those with direct property access (obj.props && obj.props.children,
etc.) and native array concat on the hot common path, leaving the rare declared
childrenProps ([]/{}) branch untouched. The crawled nodes are always plain
component objects, so this is equivalent to the curried path, just without the
dispatch and placeholder checks.
Same-machine A/B: patch_append_nested ~16% faster, initial render and wildcard
resolution a few percent, no behavior change. Renderer unit 55/55; integration
green across wildcards, basic/multiple callbacks, layout paths, patch, reorder.
Also: the benchmark profiler now keys anonymous frames by source location
(they were collapsing into one opaque bucket), and the baseline is refreshed.
T4rk1n
force-pushed
the
perf/benchmark-harness
branch
from
August 25, 2026 14:41
e429cb1 to
4cbec1d
Compare
The baseline-ratio gate compared raw milliseconds against a committed baseline, so it flaked: identical code runs ~2-4x slower on a shared CI runner than on a dev machine, tripping the >2x-baseline hard-fail on every PR even with no regression. Divide out a per-run machine scale before the baseline comparison. The runner measures a fixed calibration scenario (initial_render_small) in the same run and normalizes each scenario's baseline ratio by this-run-calibration-median / baseline-calibration-median, so a uniformly Nx-slower machine reports ~1.0x while a genuine regression still shows through. The absolute warn_ms/fail_ms ceilings stay un-scaled on purpose: generous order-of-magnitude guards that also backstop a global slowdown which would otherwise hide inside the scale. Subset runs without the calibration scenario fall back to a raw-ms comparison and say so; the detected scale is printed in the summary/PR comment. baseline.json stays committed and can now be regenerated on any machine.
The keyboard-navigation a11y tests read document.activeElement.textContent synchronously right after each keypress, but focus is moved inside a requestAnimationFrame callback. React 19's scheduling widened that window, making test_a11y003 flaky (different wrong value each run, React-19 only). Replace the immediate asserts with a wait_for_focused_text helper that polls until the focused text settles.
graph_ms baselines are sub-millisecond, so the machine-normalized baseline ratio is dominated by browser timer jitter: a single slow sample (0.9ms baseline vs 2.6ms) reads as a 1.4x 'regression' and warns on every run. Skip the baseline-ratio check when the baseline p90 is below MIN_BASELINE_MS (5ms); the absolute warn_ms/fail_ms ceilings still guard those metrics. Larger metrics (render/callback/patch, tens-to-thousands of ms) are unaffected.
A dispatch with regenerate_baseline=true measures a fresh baseline on the ubuntu-latest runner and opens a PR updating benchmarks/baseline.json. Run it on the default branch after a merge so the machine scale is ~1.0x for the next PR's gate (the current committed baseline was captured on a ~2x faster laptop). The regeneration runs as a separate job gated to the dispatch input, with its own contents:write/pull-requests:write; the normal gating job keeps read-only.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3948 (base branch
perf/patch-append-rehydration), so the diff here is only the benchmark work and the two optimizations it surfaced. Review/merge #3948 first.What's here
1. A platform performance benchmark harness (
benchmarks/)Standalone, not part of the pytest suite — timing is noisy, so it reports rather than flaking the test matrix. 10 scenarios across the platform (initial/deep hydration, Patch append top-level + nested, scalar Patch update, full-list replacement, callback fan-out,
ALL-wildcard resolution, deep callback chain). Each runs in its own subprocess against the production bundle, driven by headless Chrome, timed with in-pageperformance.now(), aggregated as median/p90/max plus agrowthratio that flags O(total) creep.--profile <scenario>captures a Chrome CPU profile + a hottest-functions table.2. CI job (
.github/workflows/benchmarks.yml)Runs on PRs touching
dash/,benchmarks/, or components. Builds the production renderer, runs vs the committedbaseline.json, hard-fails only on an order-of-magnitude regression, warns without failing on smaller drift, and upserts a sticky PR comment with the results table.3. Wildcard resolution O(n²) → O(1) — found by the harness
getPathfor a pattern-matching (dict) id did a linear deep-equality scan over every component sharing the id's key set, called once per resolved component — so anALLdispatch over N components was O(N²). Added anobjIndexhash map (valuesKey → path) maintained inline incomputePaths(copy-on-write) andappendPaths; the orderedobjsarray stays for pattern iteration. ~580ms → ~140ms (≈4x) onALLover 400 components.4. Ramda currying overhead in the per-node layout crawl — also found by the harness
crawlLayout(run on every component on every path recompute + callback gather) used curriedpath/pathOrper node. Replaced with direct property access on the hot common path. Same-machine A/B: patch_append ~16%, initial render + wildcard a few %. No behavior change..ai/PERFORMANCE.mddocuments the methodology, how to profile, and the findings (including two that are not fixed here:callback_chainis network-bound, and layouts deeper than ~250 fail to serialize).Testing
test_wildcards,test_basic_callback,test_multiple_callbacks,test_layout_paths_with_callbacks,test_patch,test_children_reorder.Note
baseline.jsonwas generated locally; absolute thresholds work anywhere, but the baseline-ratio comparison is best regenerated onubuntu-latest(adopt the first CI run'sresults.json).