Add examples/python_fraction: measuring the fraction of run time spent in Python

[DLWoodruff]

Adds examples/python_fraction/: an experiment suite and a short writeup measuring
what fraction of an mpi-sppy run is spent in Python as opposed to compiled code
(solver, numpy, MPI). Nothing outside that directory is touched, so there is no
effect on the library.

What is in it

• run_experiments.bash — runs each case under scalene,
  repeating each case so the writeup can show run-to-run spread rather than a single
  sample. PROFILE=0 reruns the same cases unprofiled, which is what the overhead
  column needs.
• scalene_wrapper.bash — rank-aware launcher so mpiexec gives each rank its own
  profile file.
• scalene_totals.py — pulls the Python/native/system split out of the scalene JSON.
• summarize_reps.py, make_scalene_latex_table.py, make_tables.bash — aggregate the
  repetitions into the LaTeX tables; make_tables.bash regenerates tables from profiles
  already on disk without rerunning anything.
• python_fraction.tex plus the two generated table files — the writeup. The built
  python_fraction.pdf is included as well, so the results can be read without a LaTeX
  toolchain.
• readme.rst — how to run it.

Every case runs the same three cylinders (PH hub, lagrangian spoke, xhatshuffle spoke,
one rank each) with the convergence tolerance set to zero, so runs end on the iteration
limit and only the model and instance change between cases.

What it found

There is no single number: the fraction depends almost entirely on how much work the
solver does per subproblem. With gurobi_persistent, sslp_15_45 sits at 8.7–9.6%
Python, while every case with small subproblems sits far above that, up to 76% for
farmer240. sslp_5_25_50 has more scenarios than sslp_15_45_15 and yet spends six
times the fraction of its time in Python, so what matters is the size of each subproblem
and not how many there are.

The mechanism is visible in the line attribution: each solve costs a roughly fixed amount
of Python work (rebuild the proximal objective, hand it to the solver, read the solution
back) that is amortized over the solver's work.

Two secondary results:

• Bundling should be judged on wall time, not on the Python fraction. It cut wall time in
  both cases tried (farmer240 82.9s → 20.9s; sslp_15_45_15 104.8s → 73.7s) but moved the
  Python share down in the first and up in the second.
• A non-persistent solver interface adds a lot of per-solve Python work: sslp_5_25_50
  goes from 58.1% to 75.4% Python, and even the solver-bound sslp_15_45_10, whose wall
  time barely moves, goes from 8.7% to 31.8%.

Caveats stated in the writeup

Scalene's overhead falls mostly on Python, so the reported Python percentages are upper
bounds; the overhead column tracks this (0.99–1.11x for the sslp cases, 1.42–1.55x for
the farmer cases). These are Pyomo models and Pyomo time counts as Python, so a good deal
of what is labelled Python is Pyomo rather than mpi-sppy. One machine, one solver, three
repetitions per case.

The suite is likely to be fragile across scalene releases, since scalene changes its
output format; it was last run against scalene 2.0.1.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.79%. Comparing base (1102c6b) to head (5faa570).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #825      +/-   ##
==========================================
+ Coverage   76.49%   76.79%   +0.30%     
==========================================
  Files         175      177       +2     
  Lines       23236    23456     +220     
==========================================
+ Hits        17774    18014     +240     
+ Misses       5462     5442      -20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a self-contained experiment suite under examples/python_fraction/ to quantify how much end-to-end mpi-sppy run time is spent in Python vs native/system code using Scalene, along with scripts to run repeated trials and generate LaTeX tables plus a short writeup (including a prebuilt PDF).

Changes:

• Added profiling runner scripts (run_experiments.bash, scalene_wrapper.bash) and table regeneration (make_tables.bash).
• Added Python utilities to extract/aggregate Scalene JSON results into LaTeX tables (scalene_totals.py, summarize_reps.py, make_scalene_latex_table.py).
• Added writeup and generated outputs (python_fraction.tex, python_fraction.pdf, scalene_summary_*.tex) plus local .gitignore.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
examples/python_fraction/summarize_reps.py	Aggregates repeated runs into summary + per-rank LaTeX tables.
examples/python_fraction/scalene_wrapper.bash	MPI-rank-aware Scalene launcher producing per-rank JSON profiles.
examples/python_fraction/scalene_totals.py	Extracts Python/native/system totals from Scalene JSON profiles with a consistency check.
examples/python_fraction/scalene_summary_persistent.tex	Generated LaTeX tables for the persistent solver interface runs.
examples/python_fraction/scalene_summary_file_interface.tex	Generated LaTeX tables for the file-based solver interface comparison runs.
examples/python_fraction/run_experiments.bash	Runs the experiment matrix with retries and optional unprofiled wall-time runs.
examples/python_fraction/readme.rst	How to run the experiments and regenerate tables.
examples/python_fraction/python_fraction.tex	Main writeup describing method, results, and caveats.
examples/python_fraction/python_fraction.pdf	Prebuilt PDF of the writeup for readers without a LaTeX toolchain.
examples/python_fraction/make_tables.bash	Regenerates the included LaTeX table files from on-disk profiles.
examples/python_fraction/make_scalene_latex_table.py	Generates a LaTeX summary for a single run directory of rank profiles.
examples/python_fraction/.gitignore	Avoids committing large profiles and common scratch artifacts.

Suppressed comments (5)

examples/python_fraction/make_tables.bash:45

• The conditional second invocation also uses python (see above). Switch to python3 for the same portability reasons.

if [[ -d "${RESULTS}/gurobi" ]]; then
  python "${HERE}/summarize_reps.py" \
    --results "${RESULTS}" \
    --solvers gurobi \
    --cases "${CASES[@]}" \
    --rank-labels "${LABELS}" \
    --out "${HERE}/scalene_summary_file_interface.tex"

examples/python_fraction/make_scalene_latex_table.py:305

• This sysctl call is executed via bash -lc only to apply 2>/dev/null. Because _run_cmd already routes stderr to DEVNULL, call sysctl directly to avoid depending on bash/shell redirection semantics.

        cpu_model = _run_cmd(["bash", "-lc", "sysctl -n machdep.cpu.brand_string 2>/dev/null"]) or None

examples/python_fraction/make_scalene_latex_table.py:316

• Same issue as above: avoid bash -lc for this sysctl query and call it directly.

        pc = _run_cmd(["bash", "-lc", "sysctl -n hw.physicalcpu 2>/dev/null"])
        if pc and pc.isdigit():

examples/python_fraction/make_scalene_latex_table.py:348

• Avoid bash -lc for this sysctl call; _run_cmd already handles stderr suppression, so direct invocation is simpler and more portable.

        mt = _run_cmd(["bash", "-lc", "sysctl -n hw.memsize 2>/dev/null"])

examples/python_fraction/make_scalene_latex_table.py:380

• _run_scalene_view_cli shells out to python, which may not be the same interpreter/venv running this script (and may not have scalene installed). Using sys.executable keeps it consistent with the current environment.

    cmd = ["python", "-m", "scalene", "view", "--cli"]

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.