From c90a53e43755d3b154c8737263524a91c873304e Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 10:00:19 -0700 Subject: [PATCH 01/34] Out-of-the-box auto-config: capture design (requirements + open question) Requirements-only design doc for a --out-of-the-box mode that introspects the environment and model to auto-assemble a sensible mpi-sppy run. Captures the confirmed requirements (user options win, env/model probing, 3-rank floor with EF fallback, transparency via equivalent command line + improvement advisory, proper bundling) and parks the decision-logic mechanism (expert system vs neural net vs nested ifs, guided by dated data files) as the explicit open first design question. No library code yet. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 142 +++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 doc/designs/out_of_the_box_design.md diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md new file mode 100644 index 000000000..9465525b6 --- /dev/null +++ b/doc/designs/out_of_the_box_design.md @@ -0,0 +1,142 @@ +# Out-of-the-box auto-configuration — design + +**Status:** Design only; no library code yet. Branch `outOfTheBox` +(off Pyomo/mpi-sppy `main`), on the DLWoodruff fork. Proceeding deliberately +("slowly"): requirements captured and confirmed; the core *decision-logic +mechanism* is still an open question (§5). +**Author:** dlw (captured with Claude Code assistance) +**Last updated:** 2026-06-28 + +--- + +## 0. Vocabulary + +**Out-of-the-box (OOTB)** mode: a single CLI switch (`--out-of-the-box`) that +lets a relatively naive user obtain a *sensible* mpi-sppy run with almost no +knowledge of the library's internals. The user supplies a model module (and +scenario data); mpi-sppy **introspects the environment and the model** and +**auto-assembles a defensible configuration** — algorithm, spokes, bundling, +solver — rather than requiring a hand-crafted hub/spoke command line. + +The *spirit* is to lower the barrier to entry: the newcomer effectively says +"here is my model, go," and gets a reasonable decomposition plus a clear +explanation of what was chosen and how to do better. + +--- + +## 1. Goals and non-goals + +### Goals + +1. A `--out-of-the-box` option (primary home: `generic_cylinders.py`) that + sets run options automatically. +2. **User options always win** (requirement 0). OOTB only *fills gaps*; any + option the user explicitly set is retained verbatim and never overridden. +3. **Environment + model introspection** (requirement 2). At minimum: the + model module, the MPI rank count, and which solvers are actually installed. + Where the OS / SLURM permits: core count and memory. Plus any options the + user already supplied. +4. **Floor at 3 ranks** (requirement 3): with fewer than 3 ranks there is no + useful cylinder configuration (hub + >= 2 spokes), so OOTB solves the **EF** + instead. +5. **Transparency** (requirement 4): print (a) the **equivalent explicit + command line** the choices imply, so the user can reproduce / learn from / + tweak it, and (b) a short, prioritized **"to improve, get..."** list + (e.g., a persistent solver, more ranks). The run proceeds regardless. +6. **Proper bundling is central** (requirement 5): auto-forming proper bundles + from scenario count vs. available ranks is a first-class part of the + decision, not an afterthought. +7. **Quick-start documentation** (requirement 1): OOTB becomes the + recommended on-ramp in the quick start. + +### Non-goals (initial cut) + +- Replacing the explicit cylinder command line for expert users. OOTB is an + *on-ramp*, and it deliberately emits the explicit command line it chose. +- Tuning convergence parameters to optimality. OOTB aims for *defensible*, + not *optimal*, configurations. +- (TBD, see §6) Deep per-model profiling / trial solves beyond what is needed + to pick a configuration. + +--- + +## 2. Confirmed scope decisions + +These were raised as scoping questions and confirmed by the user (2026-06-28): + +1. **Home + scope.** `generic_cylinders.py` is the primary (initial) entry + point. Reachability via `Amalgamator` is possible later but not the first + target. *(CONFIRMED as the assumed direction; revisit if Amalgamator + coverage is wanted sooner.)* +2. **What we read from the module.** OOTB learns scenario count / names (hence + two-stage vs. multistage structure) from the module. Whether OOTB may + *instantiate a scenario* to gauge size/difficulty vs. staying purely + structural is itself an open detail — see §6. +3. **"Dated data files."** Interpreted as a versioned, **date-stamped + knowledge base** of heuristics/benchmarks ("for problems like X with + resources like Y, configuration Z worked well") that *guides* the chooser + and can be refreshed over time — **not** user run-config files. +4. **"What would help" output.** Echo the equivalent command line *and* a + short prioritized "to improve, get..." list; both printed; the run proceeds + anyway. + +--- + +## 3. Inputs OOTB consults + +| Source | Items | Notes | +|---|---|---| +| User-supplied options | anything already on the command line / Config | Highest precedence; never overridden | +| Model module | scenario count/names, stage structure (2-stage vs multistage) | Structural read; trial instantiation is TBD (§6) | +| MPI environment | number of ranks | Drives EF-vs-cylinders and the 3-rank floor | +| Installed solvers | which solvers import / are licensed; persistent variants | Drives solver choice + a "get a persistent solver" hint | +| OS / SLURM (best effort) | core count, memory | OS-dependent; SLURM env vars when present | + +--- + +## 4. Outputs OOTB produces + +1. A fully-populated `Config` (or equivalent) that the normal driver path then + executes. +2. A printed **equivalent explicit command line**. +3. A printed prioritized **"to improve" advisory** list. +4. The run itself (EF if < 3 ranks, otherwise the chosen cylinder + configuration). + +--- + +## 5. OPEN DESIGN QUESTION — the decision-logic mechanism + +> This is the **first design question** and is intentionally **not yet +> decided.** Candidates raised, in no order: +> +> - **Expert system** (rules engine over facts about environment + model) +> - **Neural net** (learned mapping from features to configuration) +> - **Nested case statements / ifs** (hand-coded decision tree) +> +> Crosscutting all three: **dated data files** (§2.3) that *drive or guide* +> the decision process, so recommendations can evolve and be tuned over time +> without rewriting code. + +Evaluation criteria to apply once we take this up (placeholder; to be filled +in during the design discussion): transparency/explainability (OOTB must emit +*why* it chose what it chose), maintainability, the cold-start problem (what +do we do before we have data), how the dated data files are produced and +consumed, and testability. + +--- + +## 6. Open details (deferred) + +- May OOTB instantiate a single scenario to estimate size/difficulty, or stay + purely structural for the first cut? +- Exact bundling heuristic (scenarios per bundle vs. ranks) — depends on §5. +- How the dated data files are generated, versioned, and shipped. +- Amalgamator reachability (§2.1). + +--- + +## 7. Phased rollout (placeholder) + +To be drafted once §5 is settled. Per project convention, sizable redesigns +ship as review-sized phases, each green on its own. From 9df05b109293419d7e770fbdfae3862c795d5013 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 10:19:59 -0700 Subject: [PATCH 02/34] Out-of-the-box: resolve mechanism, add first dated policy file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the resolved decision-logic mechanism (authored declarative policy file + thin Python interpreter; no rules-engine library, no neural net) in the design doc (§5/§5.1) and ships the first dated policy file mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json. The policy is pure data with _comment fields throughout; every threshold is flagged _cold_start_guess pending benchmark data. No code consumes it yet. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 93 +++++++++++++---- .../ootb_policies/ootb_policy_2026-06-28.json | 99 +++++++++++++++++++ 2 files changed, 174 insertions(+), 18 deletions(-) create mode 100644 mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 9465525b6..7a5ef460f 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -105,24 +105,81 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): --- -## 5. OPEN DESIGN QUESTION — the decision-logic mechanism - -> This is the **first design question** and is intentionally **not yet -> decided.** Candidates raised, in no order: -> -> - **Expert system** (rules engine over facts about environment + model) -> - **Neural net** (learned mapping from features to configuration) -> - **Nested case statements / ifs** (hand-coded decision tree) -> -> Crosscutting all three: **dated data files** (§2.3) that *drive or guide* -> the decision process, so recommendations can evolve and be tuned over time -> without rewriting code. - -Evaluation criteria to apply once we take this up (placeholder; to be filled -in during the design discussion): transparency/explainability (OOTB must emit -*why* it chose what it chose), maintainability, the cold-start problem (what -do we do before we have data), how the dated data files are produced and -consumed, and testability. +## 5. Decision-logic mechanism (RESOLVED 2026-06-28) + +The first design question — expert system vs. neural net vs. nested case/ifs, +all optionally driven by dated data files — is **resolved**: + +**Authored, declarative, data-driven: a dated/versioned policy file holding the +knowledge, interpreted by a thin hand-written Python decision routine. No +rules-engine library, no neural net.** + +Reasoning: + +- The three candidates are really two categories: *authored* knowledge + (expert system, nested ifs — the same thing at two points on a spectrum) vs. + *learned* knowledge (neural net). The "dated data files" idea cuts across + all three and is the real commitment: **knowledge lives in data, not frozen + in code.** +- **Neural net rejected** on four independent grounds: cold start (no training + corpus, and never NN-scale data), the hard transparency requirement (must + explain *why* — req. 4), the contributor base (optimization researchers edit + rules, not models), and testability. +- **Rules-engine libraries rejected** (pyke is abandonware ~2010; experta is + stale ~2018 with `frozendict` pin problems; clipspy/CLIPS is maintained but + heaviest — C extension + its own language — for our smallest need). A RETE + engine earns its cost only with many interacting, re-firing rules; OOTB makes + a handful of decisions once, in an obvious order. An engine also adds a heavy + dependency into already-fragile MPI/solver installs and can make firing-order + *harder* to explain. +- So: implement the "expert-system framing" (facts + declarative knowledge + + thin matcher) as plain Python reading a JSON policy file (~order of 100 + lines). Conditions are **named predicates** the interpreter knows how to + evaluate — no expression language in v1. + +**Migration path preserved:** when a benchmark corpus eventually exists it +*tunes the numbers inside the next dated policy file* (e.g. a regression that +sets the EF cutoff or bundle target); structure stays rule-based and +explainable. **Escape hatch:** if the knowledge base ever explodes into dozens +of forward-chaining rules, revisit **clipspy** (never experta/pyke) by porting +the policy file's contents into a CLIPS KB — the policy-as-data design keeps +that open. We expect never to need it. + +Criteria this satisfies: transparency (every decision logs its reason → +equivalent command line + advisory fall straight out), cold-start (authored v1 +works day one), maintainability (Python + JSON), testability (deterministic +"facts X ⇒ config Y" unit tests). + +## 5.1 Policy file: location, selection, and v1 schema + +**Location.** Dated policy files ship with the library under +`mpisppy/generic/ootb_policies/` (co-located with the future OOTB code, which +belongs in the refactored `mpisppy/generic/` package — `parsing.py`, `ef.py`, +`hub.py`, `spokes.py`, `scenario_io.py`, … — not the old monolithic +`generic_cylinders.py`). First file: +`ootb_policies/ootb_policy_2026-06-28.json`. + +**Selection.** By default the interpreter loads the **newest dated file** in +that directory; a flag (proposed `--ootb-policy-file`) overrides it. The run +**logs which policy file (and `policy_version`) it used**, for reproducibility. + +**v1 schema** (see the file for the authoritative, self-documenting copy; every +threshold is flagged `_cold_start_guess` and is a placeholder to be tuned with +data): + +| Key | Purpose | +|---|---| +| `ef_fallback` | `min_ranks_for_decomposition` (=3, req. 3) and `ef_if_num_scens_at_most` | +| `solver` | `preference_order` (persistent-commercial → commercial → free QP-capable → LP/MIP-only), plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | +| `hub` | default hub factory (`ph_hub`, no flag) | +| `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | +| `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | +| `bundling` | proper-bundling targets; `--scenarios-per-bundle` must divide `num_scens`; hard `#bundles ≥ #ranks` | +| `param_defaults` | gap-fill only; minimal in v1 (`max_iterations`) | +| `advisories` | named-predicate rules → the prioritized "to improve, get…" messages (req. 4) | + +**Predicates the interpreter must implement (v1):** `ran_ef_due_to_min_ranks`, +`chosen_solver_not_persistent`, `solver_is_lp_mip_only`, `cylinders_below_max`. --- diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json new file mode 100644 index 000000000..e72efee7f --- /dev/null +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -0,0 +1,99 @@ +{ + "_description": "Out-of-the-box (OOTB) auto-configuration policy. This is DATA, not code: a thin Python interpreter reads it and a set of facts about the environment (MPI ranks, installed solvers, optional cores/memory) and the model (scenario count, stage structure) to assemble a defensible mpi-sppy run. User-supplied options always win; OOTB only fills gaps. See doc/designs/out_of_the_box_design.md.", + + "schema_version": 1, + "policy_version": "2026-06-28", + "provenance": "COLD START. Authored by hand from mpi-sppy defaults and common practice; NO benchmark data yet. Every numeric threshold below is an educated guess flagged with _cold_start_guess and is expected to be replaced by data-tuned values in a later dated policy file.", + + "ef_fallback": { + "_comment": "Requirement 3: need at least min_ranks_for_decomposition ranks, else solve the EF. Also fall back to EF when there are too few scenarios for decomposition to buy anything.", + "min_ranks_for_decomposition": 3, + "ef_if_num_scens_at_most": 2, + "_cold_start_guess": ["ef_if_num_scens_at_most"] + }, + + "solver": { + "_comment": "Picked by trying preference_order in order and keeping the first that SolverFactory(name).available(exception_flag=False) reports. Persistence is requested by NAMING the *_persistent interface (no separate flag). Persistent commercial first, then plain commercial, then free QP-capable, then LP/MIP-only (which forces prox linearization).", + "preference_order": [ + "gurobi_persistent", "cplex_persistent", "xpress_persistent", + "gurobi", "cplex", "xpress", + "appsi_highs", "highs", "ipopt", + "cbc", "glpk" + ], + "commercial": [ + "gurobi", "cplex", "xpress", + "gurobi_persistent", "cplex_persistent", "xpress_persistent" + ], + "qp_capable": [ + "gurobi", "cplex", "xpress", + "gurobi_persistent", "cplex_persistent", "xpress_persistent", + "ipopt", "highs", "appsi_highs" + ], + "lp_mip_only_force_linearize_prox": ["glpk", "cbc"], + "caveats": { + "_comment": "Surfaced via advisories, not branched on yet (model integrality is not introspected in v1; see design §6).", + "ipopt": "Continuous NLP only; cannot handle integer variables.", + "highs": "QP ok; mixed-integer quadratic (MIQP) support is limited, so the PH prox on MIP subproblems may be rejected.", + "glpk_cbc": "LP/MIP only; cannot handle the quadratic PH prox term, so OOTB sets --linearize-proximal-terms." + } + }, + + "hub": { + "_comment": "PH is the default hub in the generic driver; no flag is emitted to select it.", + "default_factory": "ph_hub", + "flag": null + }, + + "spoke_ladder": { + "_comment": "Ordered list of spokes to switch on as ranks allow, lowest priority number first. Only flags WIRED in mpisppy/generic/spokes.py appear here. The 3-rank minimum fills priorities 1 and 2 (one outer + one inner bound) so there is always an optimality gap. Each rung consumes one cylinder; see rank_allocation for how leftover ranks are spent.", + "rungs": [ + {"priority": 1, "flag": "--lagrangian", "factory": "lagrangian_spoke", "bound": "outer", "note": "Lagrangian dual bound; cheap and robust. Core."}, + {"priority": 2, "flag": "--xhatshuffle", "factory": "xhatshuffle_spoke", "bound": "inner", "note": "Incumbent finder over shuffled scenarios. Core."}, + {"priority": 3, "flag": "--fwph", "factory": "fwph_spoke", "bound": "outer", "note": "Frank-Wolfe PH; often tightens the outer bound."}, + {"priority": 4, "flag": "--xhatxbar", "factory": "xhatxbar_spoke", "bound": "inner", "note": "Cheap second incumbent source from xbar."}, + {"priority": 5, "flag": "--subgradient", "factory": "subgradient_spoke", "bound": "outer", "note": "Alternative dual bound."}, + {"priority": 6, "flag": "--reduced-costs","factory": "reduced_costs_spoke","bound": "outer", "note": "More advanced; benefits markedly from a persistent solver. Last."} + ], + "core_roster_min": { + "_comment": "Minimum spokes for a meaningful run: at least one of each bound type. With the PH hub this is the 3-rank floor.", + "outer": 1, + "inner": 1 + }, + "max_cylinders": 6, + "_cold_start_guess": ["max_cylinders", "ladder ordering of priorities 3-6"] + }, + + "rank_allocation": { + "_comment": "How to spend ranks. First fill the spoke ladder up to max_cylinders (each cylinder = 1 rung + the hub). Then, rather than adding lower-value spokes, spend remaining ranks on intra-cylinder parallelism (more ranks per cylinder => faster subproblem throughput), especially when there are many scenarios/bundles. v1 keeps this simple and conservative.", + "fill_ladder_before_intra_cylinder": true, + "extra_ranks_policy": "intra_cylinder" + }, + + "bundling": { + "_comment": "Proper bundling only (loose bundling was removed in 2026). The live flag is --scenarios-per-bundle and it MUST divide num_scens evenly. Hard constraint enforced by the library: number_of_bundles >= number_of_ranks. Strategy: when there are many scenarios, choose a scenarios_per_bundle that divides num_scens and yields about target_bundles_per_intra_rank bundles per intra-cylinder rank, never dropping below the rank count.", + "min_scens_to_consider_bundling": 50, + "target_bundles_per_intra_rank": 2, + "never_fewer_bundles_than_ranks": true, + "_cold_start_guess": ["min_scens_to_consider_bundling", "target_bundles_per_intra_rank"] + }, + + "param_defaults": { + "_comment": "Filled ONLY if the user did not set them. Kept minimal in v1 to avoid surprising the user; flag names not yet verified are intentionally omitted rather than guessed.", + "max_iterations": 100, + "_cold_start_guess": ["max_iterations"] + }, + + "advisories": { + "_comment": "Requirement 4: a prioritized 'to improve, get...' list. Each entry has a named predicate the interpreter knows how to evaluate (keeps conditions explainable; no expression language needed in v1) and a message. Printed after the run is configured; the run proceeds regardless.", + "rules": [ + {"id": "ran_ef_too_few_ranks", "when": "ran_ef_due_to_min_ranks", "priority": 1, + "message": "Ran the monolithic EF because fewer than 3 MPI ranks were available. With >= 3 ranks, mpi-sppy can run a parallel decomposition (hub + bound spokes) instead."}, + {"id": "no_persistent_solver", "when": "chosen_solver_not_persistent", "priority": 2, + "message": "The chosen solver has no persistent interface available. Installing/licensing a persistent interface (e.g. gurobi_persistent) lets subproblems warm-start and is typically much faster for PH."}, + {"id": "linearized_prox", "when": "solver_is_lp_mip_only", "priority": 3, + "message": "An LP/MIP-only solver was chosen, so the PH proximal term is being linearized (--linearize-proximal-terms). A QP-capable solver (gurobi/cplex/xpress, or ipopt for continuous models) avoids this approximation."}, + {"id": "more_ranks_help", "when": "cylinders_below_max", "priority": 4, + "message": "More MPI ranks would let OOTB add bound-tightening spokes (e.g. fwph, subgradient) and/or more intra-cylinder parallelism."} + ] + } +} From 3ea768b37583e483aa70c8b7ce3374a0d4b1e8fc Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 10:54:04 -0700 Subject: [PATCH 03/34] Out-of-the-box: add effort tiers (instantiation depth) to design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the instantiation question as a user-selectable effort axis: --out-of-the-box-minus (no instantiation), --out-of-the-box (one probe scenario; default), --out-of-the-box-plus (all + brief solve; later). One interpreter + one policy; tiers differ only in gather_facts depth, and decisions degrade gracefully to advisories when a fact is absent. Adds the "probe not reuse-all" intervention-seam rationale and a probe_scenarios knob (default 1, lands with the base tier). Marks the bundling heuristic resolved, records PR1 (pipeline + minus + base) / PR2 (plus) phasing, and reconciles the §1/§2/§3 surfaces that previously said the instantiation question was TBD. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 89 +++++++++++++++++++++++----- 1 file changed, 73 insertions(+), 16 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 7a5ef460f..598b94e17 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -55,8 +55,8 @@ explanation of what was chosen and how to do better. *on-ramp*, and it deliberately emits the explicit command line it chose. - Tuning convergence parameters to optimality. OOTB aims for *defensible*, not *optimal*, configurations. -- (TBD, see §6) Deep per-model profiling / trial solves beyond what is needed - to pick a configuration. +- Running a solver to make decisions in the default path. Trial solves are + confined to the opt-in `plus` tier (§5.2); minus and base never solve. --- @@ -69,9 +69,9 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): target. *(CONFIRMED as the assumed direction; revisit if Amalgamator coverage is wanted sooner.)* 2. **What we read from the module.** OOTB learns scenario count / names (hence - two-stage vs. multistage structure) from the module. Whether OOTB may - *instantiate a scenario* to gauge size/difficulty vs. staying purely - structural is itself an open detail — see §6. + two-stage vs. multistage structure) from the module. Whether OOTB also + *instantiates scenarios* to gauge size/difficulty is **resolved** as a + user-selectable effort axis (none / one probe / all) — see §5.2. 3. **"Dated data files."** Interpreted as a versioned, **date-stamped knowledge base** of heuristics/benchmarks ("for problems like X with resources like Y, configuration Z worked well") that *guides* the chooser @@ -87,7 +87,7 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): | Source | Items | Notes | |---|---|---| | User-supplied options | anything already on the command line / Config | Highest precedence; never overridden | -| Model module | scenario count/names, stage structure (2-stage vs multistage) | Structural read; trial instantiation is TBD (§6) | +| Model module | scenario count/names, stage structure (2-stage vs multistage); at base/plus also integrality, per-scenario size, nonant count | Depth set by the effort tier (§5.2): minus structural-only, base one probe, plus all | | MPI environment | number of ranks | Drives EF-vs-cylinders and the 3-rank floor | | Installed solvers | which solvers import / are licensed; persistent variants | Drives solver choice + a "get a persistent solver" hint | | OS / SLURM (best effort) | core count, memory | OS-dependent; SLURM env vars when present | @@ -181,19 +181,76 @@ data): **Predicates the interpreter must implement (v1):** `ran_ef_due_to_min_ranks`, `chosen_solver_not_persistent`, `solver_is_lp_mip_only`, `cylinders_below_max`. +## 5.2 Effort tiers (instantiation depth) — RESOLVED 2026-06-28 + +OOTB exposes three tiers along an **effort axis** — how deeply it inspects the +model — selected by mutually-exclusive flags. They share ONE interpreter and +ONE policy file; the tiers differ only in how deeply `gather_facts()` populates +the `Facts` object. Every decision uses the **best fact available** and +**degrades gracefully** (to structural reasoning, or to an advisory) when a +fact is absent. Requirement 0 (the user's explicit options win) is orthogonal +to the tier. + +| Tier | Flag | Instantiates | New facts | What it can decide (vs. advise) | +|---|---|---|---|---| +| minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability. Integrality/size unknown → only **advises** on prox linearization | +| base (default) | `--out-of-the-box` | **one** probe scenario | integrality, per-scenario size, nonant count | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; memory-aware bundling | +| plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | + +**Mechanism.** The three flags set one internal `ootb_effort` level +(`minus`/`base`/`plus`); at most one may be supplied. This keeps a single code +path — richer tiers merely turn advisories into decisions. + +**Why a probe, not reuse-all (the intervention seam).** The *structural* +choices — EF-vs-decomposition, bundling, cylinder/comm/rank layout — must be +fixed *before* models are built: they determine what is instantiated and the +MPI topology, and proper bundling builds a *new combined* Pyomo model, so +singleton instantiations cannot be reused as bundles (Pyomo components cannot +be re-parented). Instantiating everything first and then restructuring wastes +work and reshuffles comms. So **base** instantiates only a cheap *probe* +(discarded) for the structural decisions; the normal driver then does the real, +layout-correct instantiation — needed anyway — which also feeds any *parametric* +refinement (solver, spokes, rho, iterations) where intervening is free. +**plus** accepts more redundant work by design, in exchange for solve-time / +gap information. + +**`probe_scenarios` knob (lands with the base tier).** How many representative +scenarios base/plus instantiate (`scenario_names[:probe_scenarios]`). Default +**1** — enough for structurally homogeneous scenarios; bump it for models whose +size or integrality varies by scenario. Added to the policy file when base is +implemented (PR1); not in the 2026-06-28 file. + --- -## 6. Open details (deferred) +## 6. Open details -- May OOTB instantiate a single scenario to estimate size/difficulty, or stay - purely structural for the first cut? -- Exact bundling heuristic (scenarios per bundle vs. ranks) — depends on §5. -- How the dated data files are generated, versioned, and shipped. -- Amalgamator reachability (§2.1). +- **Instantiation depth — RESOLVED** as the effort tiers (§5.2): minus (none), + base (one probe, default), plus (all + brief solve, later). +- **Bundling heuristic — RESOLVED** in the policy (`bundling`) + interpreter + (`_pick_scenarios_per_bundle` divisor search): aim for + ~`target_bundles_per_intra_rank` bundles per intra-cylinder rank, never fewer + than the rank count, with `scenarios_per_bundle` dividing `num_scens`. The + numbers are `_cold_start_guess`es. +- **Still open:** how the dated data files are generated, versioned, and shipped + (the §5 migration path anticipates data-tuned successors). +- **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. --- -## 7. Phased rollout (placeholder) - -To be drafted once §5 is settled. Per project convention, sizable redesigns -ship as review-sized phases, each green on its own. +## 7. Phased rollout + +Per project convention, ship as review-sized phases, each green on its own. A +sketch of the interpreter already exists at `mpisppy/generic/out_of_the_box.py` +— the pure `recommend(facts, policy) → Decision` logic is complete and +smoke-tested; environment/model probing and apply-to-`Config` are stubbed. + +- **PR1 — interpreter pipeline + `--out-of-the-box-minus` + `--out-of-the-box` + (base).** These share everything except one probe instantiation, so they land + together: fact-gathering (structural + one-scenario probe), the policy + interpreter, EF-vs-cylinder / bundling / spoke selection, reporting + (equivalent command line + advisories), apply-to-`Config`, the + `probe_scenarios` knob, quick-start docs, and tests. Ships the default + on-ramp and the no-instantiation escape hatch in one PR. +- **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed + solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds + iteration/time-limit defaults and solve-cost-aware bundling. From d02ee49c87a5f02961e68257c47daa1fb009a469 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 10:56:12 -0700 Subject: [PATCH 04/34] Out-of-the-box: internal-consistency pass on the design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconcile surfaces left stale by the §5/§5.2 resolutions: - Status header no longer calls the mechanism "an open question" or claims "no library code yet" (sketch exists; policy committed). - Vocabulary: "single CLI switch" -> the three effort-tier flags. - Goal 1 + §5.1 agree: generic_cylinders.py is the CLI entry, OOTB code lives in the refactored mpisppy/generic/ package (was contradictory). - Output 4: EF gate is not only "< 3 ranks"; also too-few/too-large scenarios. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 32 ++++++++++++++++------------ 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 598b94e17..c1712a5f1 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -1,9 +1,10 @@ # Out-of-the-box auto-configuration — design -**Status:** Design only; no library code yet. Branch `outOfTheBox` -(off Pyomo/mpi-sppy `main`), on the DLWoodruff fork. Proceeding deliberately -("slowly"): requirements captured and confirmed; the core *decision-logic -mechanism* is still an open question (§5). +**Status:** Design phase. Branch `outOfTheBox` (off Pyomo/mpi-sppy `main`), on +the DLWoodruff fork. Proceeding deliberately ("slowly"): requirements +confirmed; the decision-logic mechanism (§5) and instantiation effort tiers +(§5.2) are resolved; the first dated policy file is committed. No production +library code yet — only an uncommitted interpreter *sketch* (§7). **Author:** dlw (captured with Claude Code assistance) **Last updated:** 2026-06-28 @@ -11,9 +12,10 @@ mechanism* is still an open question (§5). ## 0. Vocabulary -**Out-of-the-box (OOTB)** mode: a single CLI switch (`--out-of-the-box`) that -lets a relatively naive user obtain a *sensible* mpi-sppy run with almost no -knowledge of the library's internals. The user supplies a model module (and +**Out-of-the-box (OOTB)** mode: a CLI switch (`--out-of-the-box`, with a lighter +`--out-of-the-box-minus` and a heavier `--out-of-the-box-plus` variant — §5.2) +that lets a relatively naive user obtain a *sensible* mpi-sppy run with almost +no knowledge of the library's internals. The user supplies a model module (and scenario data); mpi-sppy **introspects the environment and the model** and **auto-assembles a defensible configuration** — algorithm, spokes, bundling, solver — rather than requiring a hand-crafted hub/spoke command line. @@ -28,8 +30,9 @@ explanation of what was chosen and how to do better. ### Goals -1. A `--out-of-the-box` option (primary home: `generic_cylinders.py`) that - sets run options automatically. +1. A `--out-of-the-box` option (CLI entry `generic_cylinders.py`, implemented + in the refactored `mpisppy/generic/` package) that sets run options + automatically. 2. **User options always win** (requirement 0). OOTB only *fills gaps*; any option the user explicitly set is retained verbatim and never overridden. 3. **Environment + model introspection** (requirement 2). At minimum: the @@ -100,7 +103,8 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): executes. 2. A printed **equivalent explicit command line**. 3. A printed prioritized **"to improve" advisory** list. -4. The run itself (EF if < 3 ranks, otherwise the chosen cylinder +4. The run itself (EF when the EF gate trips — too few ranks, or too few / + too large scenarios per §5.1/§5.2 — otherwise the chosen cylinder configuration). --- @@ -153,10 +157,10 @@ works day one), maintainability (Python + JSON), testability (deterministic ## 5.1 Policy file: location, selection, and v1 schema **Location.** Dated policy files ship with the library under -`mpisppy/generic/ootb_policies/` (co-located with the future OOTB code, which -belongs in the refactored `mpisppy/generic/` package — `parsing.py`, `ef.py`, -`hub.py`, `spokes.py`, `scenario_io.py`, … — not the old monolithic -`generic_cylinders.py`). First file: +`mpisppy/generic/ootb_policies/` (co-located with the OOTB code, which lives in +the refactored `mpisppy/generic/` package — `parsing.py`, `ef.py`, `hub.py`, +`spokes.py`, `scenario_io.py`, … ; `generic_cylinders.py` remains the +user-facing CLI entry that delegates into this package). First file: `ootb_policies/ootb_policy_2026-06-28.json`. **Selection.** By default the interpreter loads the **newest dated file** in From 0a49efbe7383dc7c372fad788e4a1a514a30abfa Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 11:14:10 -0700 Subject: [PATCH 05/34] Out-of-the-box: policy selection is an optional path on each flag Each effort flag takes an optional policy-file path (domain=str, default=None, argparse_args={'nargs':'?','const':}): absent -> off, bare -> shipped default (newest dated file with no focus token), PATH -> that file. Drops the separate --ootb-policy-file flag, focus-name resolution, and any machine-read focus field; multiple foci ship as differently-named files and the filename documents the focus for humans. Notes the bool-domain caveat (pyomo forces store_true for bool, so optional values require a str domain; add_to_config forwards argparse_args to add_argument). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index c1712a5f1..73a045a75 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -163,8 +163,15 @@ the refactored `mpisppy/generic/` package — `parsing.py`, `ef.py`, `hub.py`, user-facing CLI entry that delegates into this package). First file: `ootb_policies/ootb_policy_2026-06-28.json`. -**Selection.** By default the interpreter loads the **newest dated file** in -that directory; a flag (proposed `--ootb-policy-file`) overrides it. The run +**Selection.** Each OOTB flag takes an **optional policy-file path** (see the +§5.2 mechanism): a bare flag uses the shipped default (the newest dated file +whose name carries no focus token, e.g. `ootb_policy_.json`); +`--out-of-the-box PATH` uses `PATH`. There is **no separate policy-file flag** — +the optional value *is* the path. Multiple policy files with **different foci** +may ship side by side; the **focus is conveyed by the filename** (e.g. +`ootb_policy_quick_.json`), which is human-facing documentation only — the +program does not parse focus from the name, and no machine-read `focus` field is +needed. A user selects a focus simply by passing that file's path. The run **logs which policy file (and `policy_version`) it used**, for reproducibility. **v1 schema** (see the file for the authoritative, self-documenting copy; every @@ -202,8 +209,15 @@ to the tier. | plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | **Mechanism.** The three flags set one internal `ootb_effort` level -(`minus`/`base`/`plus`); at most one may be supplied. This keeps a single code -path — richer tiers merely turn advisories into decisions. +(`minus`/`base`/`plus`); at most one may be supplied. Each takes an **optional +policy-file path** — declared `domain=str, default=None, +argparse_args={'nargs': '?', 'const': }` — giving three states: +absent → `None` → OOTB off; bare flag → `const` → the default policy; `PATH` → +that policy file. (A `bool` domain cannot be used: pyomo's +`declare_as_argument` forces `store_true` for bool, which takes no value; +`str` + `nargs='?'` is the supported optional-value form, and `add_to_config` +forwards `argparse_args` straight to `argparse.add_argument`.) This keeps a +single code path — richer tiers merely turn advisories into decisions. **Why a probe, not reuse-all (the intervention seam).** The *structural* choices — EF-vs-decomposition, bundling, cylinder/comm/rank layout — must be From c43db0ef29236e83fa565a43e27ffa2cdd8df868 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 11:26:27 -0700 Subject: [PATCH 06/34] Out-of-the-box: rename advisories -> suggestions, emit after the run The "to improve" output is relabelled "Suggestions" and is written after the algorithm executes (so it can also reflect how the run went), with the equivalent command line still printed up front. Renames the policy key advisories -> suggestions and reconciles the design doc (goals, outputs, schema, tiers, phasing). Interpreter sketch (uncommitted) follows suit: make_suggestions(decision, facts, policy, outcome=None) is a post-run step, recommend() no longer computes them. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 30 ++++++++++--------- .../ootb_policies/ootb_policy_2026-06-28.json | 6 ++-- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 73a045a75..fc0c052b0 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -44,8 +44,10 @@ explanation of what was chosen and how to do better. instead. 5. **Transparency** (requirement 4): print (a) the **equivalent explicit command line** the choices imply, so the user can reproduce / learn from / - tweak it, and (b) a short, prioritized **"to improve, get..."** list - (e.g., a persistent solver, more ranks). The run proceeds regardless. + tweak it, and (b) a short, prioritized list of **suggestions** (labelled + "Suggestions"), **written after the run executes** so it can also reflect + how the run went (e.g., a persistent solver, more ranks). The run proceeds + regardless. 6. **Proper bundling is central** (requirement 5): auto-forming proper bundles from scenario count vs. available ranks is a first-class part of the decision, not an afterthought. @@ -80,8 +82,8 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): resources like Y, configuration Z worked well") that *guides* the chooser and can be refreshed over time — **not** user run-config files. 4. **"What would help" output.** Echo the equivalent command line *and* a - short prioritized "to improve, get..." list; both printed; the run proceeds - anyway. + short prioritized **Suggestions** list — the command line up front, the + suggestions after the run; the run proceeds anyway. --- @@ -101,11 +103,11 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): 1. A fully-populated `Config` (or equivalent) that the normal driver path then executes. -2. A printed **equivalent explicit command line**. -3. A printed prioritized **"to improve" advisory** list. -4. The run itself (EF when the EF gate trips — too few ranks, or too few / - too large scenarios per §5.1/§5.2 — otherwise the chosen cylinder - configuration). +2. A printed **equivalent explicit command line** (up front, before the run). +3. The run itself (EF when the EF gate trips — too few ranks, or the problem is + small enough per §5.1/§5.2 — otherwise the chosen cylinder configuration). +4. A printed prioritized **Suggestions** list, emitted **after the run** + (labelled "Suggestions"; may reflect how the run went). --- @@ -150,7 +152,7 @@ the policy file's contents into a CLIPS KB — the policy-as-data design keeps that open. We expect never to need it. Criteria this satisfies: transparency (every decision logs its reason → -equivalent command line + advisory fall straight out), cold-start (authored v1 +equivalent command line + suggestions fall straight out), cold-start (authored v1 works day one), maintainability (Python + JSON), testability (deterministic "facts X ⇒ config Y" unit tests). @@ -187,7 +189,7 @@ data): | `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | | `bundling` | proper-bundling targets; `--scenarios-per-bundle` must divide `num_scens`; hard `#bundles ≥ #ranks` | | `param_defaults` | gap-fill only; minimal in v1 (`max_iterations`) | -| `advisories` | named-predicate rules → the prioritized "to improve, get…" messages (req. 4) | +| `suggestions` | named-predicate rules → the prioritized **Suggestions** list (req. 4), emitted after the run | **Predicates the interpreter must implement (v1):** `ran_ef_due_to_min_ranks`, `chosen_solver_not_persistent`, `solver_is_lp_mip_only`, `cylinders_below_max`. @@ -198,7 +200,7 @@ OOTB exposes three tiers along an **effort axis** — how deeply it inspects the model — selected by mutually-exclusive flags. They share ONE interpreter and ONE policy file; the tiers differ only in how deeply `gather_facts()` populates the `Facts` object. Every decision uses the **best fact available** and -**degrades gracefully** (to structural reasoning, or to an advisory) when a +**degrades gracefully** (to structural reasoning, or to a suggestion) when a fact is absent. Requirement 0 (the user's explicit options win) is orthogonal to the tier. @@ -217,7 +219,7 @@ that policy file. (A `bool` domain cannot be used: pyomo's `declare_as_argument` forces `store_true` for bool, which takes no value; `str` + `nargs='?'` is the supported optional-value form, and `add_to_config` forwards `argparse_args` straight to `argparse.add_argument`.) This keeps a -single code path — richer tiers merely turn advisories into decisions. +single code path — richer tiers merely turn suggestions into decisions. **Why a probe, not reuse-all (the intervention seam).** The *structural* choices — EF-vs-decomposition, bundling, cylinder/comm/rank layout — must be @@ -266,7 +268,7 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. (base).** These share everything except one probe instantiation, so they land together: fact-gathering (structural + one-scenario probe), the policy interpreter, EF-vs-cylinder / bundling / spoke selection, reporting - (equivalent command line + advisories), apply-to-`Config`, the + (equivalent command line + post-run suggestions), apply-to-`Config`, the `probe_scenarios` knob, quick-start docs, and tests. Ships the default on-ramp and the no-instantiation escape hatch in one PR. - **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index e72efee7f..d811fe703 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -31,7 +31,7 @@ ], "lp_mip_only_force_linearize_prox": ["glpk", "cbc"], "caveats": { - "_comment": "Surfaced via advisories, not branched on yet (model integrality is not introspected in v1; see design §6).", + "_comment": "Surfaced via suggestions, not branched on yet (model integrality is not introspected in v1; see design §6).", "ipopt": "Continuous NLP only; cannot handle integer variables.", "highs": "QP ok; mixed-integer quadratic (MIQP) support is limited, so the PH prox on MIP subproblems may be rejected.", "glpk_cbc": "LP/MIP only; cannot handle the quadratic PH prox term, so OOTB sets --linearize-proximal-terms." @@ -83,8 +83,8 @@ "_cold_start_guess": ["max_iterations"] }, - "advisories": { - "_comment": "Requirement 4: a prioritized 'to improve, get...' list. Each entry has a named predicate the interpreter knows how to evaluate (keeps conditions explainable; no expression language needed in v1) and a message. Printed after the run is configured; the run proceeds regardless.", + "suggestions": { + "_comment": "Requirement 4: a prioritized 'Suggestions' list, printed AFTER the run executes (so it can also reflect how the run went). Each entry has a named predicate the interpreter evaluates (explainable; no expression language in v1) and a message; the run proceeds regardless.", "rules": [ {"id": "ran_ef_too_few_ranks", "when": "ran_ef_due_to_min_ranks", "priority": 1, "message": "Ran the monolithic EF because fewer than 3 MPI ranks were available. With >= 3 ranks, mpi-sppy can run a parallel decomposition (hub + bound spokes) instead."}, From 90c5a117ca8c01a284eb06547b6380b60d7c6a92 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 11:53:14 -0700 Subject: [PATCH 07/34] Out-of-the-box: bundle sizing via a shared effort-scaling model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundle size becomes a derived quantity, not a specified one: pick the largest scenarios_per_bundle that divides num_scens, leaves >= B_min bundles, and keeps a bundle's modeled solve effort within budget. Adds a shared effort_scaling shape (continuous ~linear, integers superlinear via int_exponent, integer nonants a fixed coupling cost) and a bundle_sizing block. Two anchors on one shape: base uses a relative budget (effort vs a single scenario, unit-free), plus uses the same shape calibrated by a measured single-scenario solve time; minus falls back to a count target. New §5.3 documents it, including why measurement (plus) calibrates but does not remove the JSON shape (MIP times are noisy), and why the EF gate needs an absolute ceiling (so it stays count-based for now). Replaces the old bundling/target_bundles heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 76 +++++++++++++++++-- .../ootb_policies/ootb_policy_2026-06-28.json | 21 ++++- 2 files changed, 86 insertions(+), 11 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index fc0c052b0..3670d6231 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -187,7 +187,8 @@ data): | `hub` | default hub factory (`ph_hub`, no flag) | | `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | | `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | -| `bundling` | proper-bundling targets; `--scenarios-per-bundle` must divide `num_scens`; hard `#bundles ≥ #ranks` | +| `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | +| `bundle_sizing` | how big bundles are: largest `spb` within an effort budget (base = relative M; plus = measured seconds; minus = count); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | | `param_defaults` | gap-fill only; minimal in v1 (`max_iterations`) | | `suggestions` | named-predicate rules → the prioritized **Suggestions** list (req. 4), emitted after the run | @@ -207,7 +208,7 @@ to the tier. | Tier | Flag | Instantiates | New facts | What it can decide (vs. advise) | |---|---|---|---|---| | minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability. Integrality/size unknown → only **advises** on prox linearization | -| base (default) | `--out-of-the-box` | **one** probe scenario | integrality, per-scenario size, nonant count | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; memory-aware bundling | +| base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int` | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; effort-budgeted bundle sizing (§5.3) | | plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | **Mechanism.** The three flags set one internal `ootb_effort` level @@ -240,17 +241,78 @@ scenarios base/plus instantiate (`scenario_names[:probe_scenarios]`). Default size or integrality varies by scenario. Added to the policy file when base is implemented (PR1); not in the 2026-06-28 file. +## 5.3 Bundle sizing & effort scaling — how big, not just whether + +The hard question is not *whether* to bundle but *how big* bundles should be — +unportable in raw variable counts (the same count is trivial for one model, +intractable for another). The design makes bundle size a **derived** quantity: +pick the **largest** `scenarios_per_bundle` (`spb`) that (a) divides +`num_scens`, (b) leaves at least `B_min = max(intra_ranks, +min_bundles_per_intra_rank · intra_ranks)` bundles, and (c) keeps a bundle's +**modeled solve effort** within a budget. + +**Shared effort shape (`effort_scaling`).** One policy block models how solve +effort grows with sub-problem size, from the probe profile (`vars_cont`, +`vars_int`, `nonants_int`): + +> `effort(spb) = cont_coeff·(spb·vars_cont) + int_weight·(spb·vars_int)^int_exponent + int_nonant_coeff·nonants_int` + +Continuous content is ~linear; integers are **superlinear** (`int_exponent > 1` +captures branch-and-bound blow-up); integer nonants are a fixed per-bundle +coupling cost. (Second-stage integers scale with `spb`; integer *nonants* are +first-stage, shared once per bundle — hence a fixed term, not a `spb` +multiplier.) + +**Two anchors, one shape.** Both tiers call the same `effort(spb)`; only the +budget differs: + +- **base** (relative, no measurement): accept the largest `spb` with + `effort(spb)/effort(1) ≤ base_max_hardness_vs_single_scenario` (**M**). M is + unit-free and portable — "a bundle may be at most M× as hard as one + scenario." Pure-continuous ⇒ allowed `spb ≈ M`; pure-integer ⇒ + `≈ M^(1/int_exponent)`, automatically smaller. +- **plus** (absolute, measured): measure `t₁` = a single-scenario solve (capped + at `plus_probe_solve_time_cap_seconds`), predict + `t(spb) ≈ t₁·effort(spb)/effort(1)`, accept the largest `spb` with + `t(spb) ≤ plus_target_seconds_per_bundle`. +- **minus** (no profile): effort is unknown → fall back to a conservative count + target (`fallback_bundles_per_intra_rank` bundles per intra-rank). + +**Measurement does not remove the JSON assumptions.** `plus` only pins the +*scale* (`t₁`); the *shape* (`int_exponent`, weights) still comes from +`effort_scaling`. And MIP solve times are **noisy and non-monotone** (a bigger +MIP can solve faster), so a single timing must not drive the whole choice — the +JSON shape is a **prior/regularizer** that measurement calibrates. (Later +refinement: `plus` measures two points to nudge `int_exponent` locally.) + +**EF gate vs. bundle sizing.** Both use the same effort *shape*, but the EF gate +needs an **absolute** ceiling ("is the monolith small enough to solve as one +model?"); the relative-to-a-single-scenario trick does not gate the whole EF. +That absolute threshold is the genuinely hard part without measurement, so the +EF gate stays **count/rank-based for now** (§5.1 `ef_fallback`) — reusing +`effort` with an absolute ceiling (or deferring to `plus`) is a follow-on. + +**Status.** All `effort_scaling` / `bundle_sizing` numbers are +`_cold_start_guess`es; **foci** ship different shapes (a `mip-heavy` file with a +steeper `int_exponent`), and the dated-file migration path (§5) refines the +coefficients from benchmark data. The interpreter sketch implements the base +relative sizer (`_effort`, `_pick_spb_by_effort`) with `_pick_spb_by_count` as +the minus fallback; the `plus` measure-and-scale hook is stubbed. + --- ## 6. Open details - **Instantiation depth — RESOLVED** as the effort tiers (§5.2): minus (none), base (one probe, default), plus (all + brief solve, later). -- **Bundling heuristic — RESOLVED** in the policy (`bundling`) + interpreter - (`_pick_scenarios_per_bundle` divisor search): aim for - ~`target_bundles_per_intra_rank` bundles per intra-cylinder rank, never fewer - than the rank count, with `scenarios_per_bundle` dividing `num_scens`. The - numbers are `_cold_start_guess`es. +- **Bundle sizing — RESOLVED** as an effort-budgeted rule (§5.3): policy + `effort_scaling` shape + `bundle_sizing` budgets; interpreter `_effort` / + `_pick_spb_by_effort` (base, relative M) with `_pick_spb_by_count` (minus + fallback) and a stubbed `plus` measure-and-scale. Numbers are + `_cold_start_guess`es. +- **Still open:** an absolute, size-aware **EF gate** — reuse the effort *shape* + with an absolute ceiling (unlike bundle sizing's relative budget) or defer to + `plus`; count/rank-based for now (§5.3). - **Still open:** how the dated data files are generated, versioned, and shipped (the §5 migration path anticipates data-tuned successors). - **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index d811fe703..d128c8771 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -69,12 +69,25 @@ "extra_ranks_policy": "intra_cylinder" }, - "bundling": { - "_comment": "Proper bundling only (loose bundling was removed in 2026). The live flag is --scenarios-per-bundle and it MUST divide num_scens evenly. Hard constraint enforced by the library: number_of_bundles >= number_of_ranks. Strategy: when there are many scenarios, choose a scenarios_per_bundle that divides num_scens and yields about target_bundles_per_intra_rank bundles per intra-cylinder rank, never dropping below the rank count.", + "effort_scaling": { + "_comment": "SHAPE of how solve effort grows with the size of a (sub)problem. Shared by bundle sizing at every tier (design doc Bundle sizing section). effort(n) for a bundle of n scenarios = cont_coeff*(n*vars_cont) + int_weight*(n*vars_int)^int_exponent + int_nonant_coeff*nonants_int, using the probe size profile (vars_cont, vars_int, nonants_int). The base tier uses this RELATIVE (ratio vs a single scenario); the plus tier uses the SAME shape CALIBRATED by a measured single-scenario solve time. int_exponent>1 captures MIP superlinearity; =1 treats integers like LP. All cold-start guesses; foci ship different shapes; benchmark data refines.", + "cont_coeff": 1.0, + "int_weight": 10.0, + "int_exponent": 2.0, + "int_nonant_coeff": 5.0, + "_cold_start_guess": ["cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff"] + }, + + "bundle_sizing": { + "_comment": "How big should proper bundles be (not just whether to bundle)? Proper bundling only (loose bundling removed in 2026); --scenarios-per-bundle MUST divide num_scens; library requires number_of_bundles >= number_of_ranks. Rule: pick the LARGEST scenarios_per_bundle that (a) divides num_scens, (b) leaves >= max(intra_ranks, min_bundles_per_intra_rank*intra_ranks) bundles, and (c) keeps a bundle's modeled effort within budget. base budget is RELATIVE: effort(spb)/effort(1) <= base_max_hardness_vs_single_scenario (unitless, portable, no measurement). plus budget is ABSOLUTE: measured_single_scenario_seconds * effort(spb)/effort(1) <= plus_target_seconds_per_bundle, with the measuring solve capped at plus_probe_solve_time_cap_seconds. Without a size profile (minus tier) effort is unknown, so fall back to fallback_bundles_per_intra_rank bundles per rank. Skip bundling below min_scens_to_consider_bundling. MIP solve times are noisy/non-monotone, so the shape in effort_scaling acts as a prior; a single measurement only sets the scale.", "min_scens_to_consider_bundling": 50, - "target_bundles_per_intra_rank": 2, + "min_bundles_per_intra_rank": 1, + "fallback_bundles_per_intra_rank": 2, "never_fewer_bundles_than_ranks": true, - "_cold_start_guess": ["min_scens_to_consider_bundling", "target_bundles_per_intra_rank"] + "base_max_hardness_vs_single_scenario": 10, + "plus_target_seconds_per_bundle": 30, + "plus_probe_solve_time_cap_seconds": 60, + "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "fallback_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] }, "param_defaults": { From c364aec3e1e2c46fb2f38a42a64e187f796cd1de Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 12:34:19 -0700 Subject: [PATCH 08/34] Out-of-the-box: add --inspect-only (dry run) to the design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A general driver flag (plain store_true, not OOTB-specific): do the inspection, print the configuration + equivalent command line + config-time suggestions, then stop before the production run. Semantics are "no production run," not "no solver call ever": per the chosen resolution B, out-of-the-box-plus's brief calibration solves count as inspection, so plus + --inspect-only still measures then stops. Used standalone (no --out-of-the-box) it verifies a scenario can be instantiated, reusing the OOTB probe via a shared verify_instantiation. With --out-of-the-box-minus it is silly but allowed, with --inspect-only taking priority (one verification instantiation; decision stays minus-level). New §5.4; folded into PR1 scope. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 31 ++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 3670d6231..267307d33 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -105,7 +105,8 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): executes. 2. A printed **equivalent explicit command line** (up front, before the run). 3. The run itself (EF when the EF gate trips — too few ranks, or the problem is - small enough per §5.1/§5.2 — otherwise the chosen cylinder configuration). + small enough per §5.1/§5.2 — otherwise the chosen cylinder configuration); + **skipped entirely under `--inspect-only`** (§5.4). 4. A printed prioritized **Suggestions** list, emitted **after the run** (labelled "Suggestions"; may reflect how the run went). @@ -299,6 +300,31 @@ coefficients from benchmark data. The interpreter sketch implements the base relative sizer (`_effort`, `_pick_spb_by_effort`) with `_pick_spb_by_count` as the minus fallback; the `plus` measure-and-scale hook is stubbed. +## 5.4 `--inspect-only` (dry run; shares OOTB's instantiation) — RESOLVED 2026-06-28 + +A general driver flag (not OOTB-specific, but documented here because it shares +code): do the inspection, **print the configuration + equivalent command line + +config-time suggestions, then stop before the production optimization run.** + +- **Semantics:** "no *production* run," not "no solver call ever." Per the + user's decision, **`--out-of-the-box-plus`'s brief calibration solves count as + inspection** (resolution B) — they are bounded by + `plus_probe_solve_time_cap_seconds` and are *how* `plus` forms its + recommendation — so `plus` + `--inspect-only` still measures, then stops. +- **Standalone (no `--out-of-the-box`):** `--inspect-only` **verifies a scenario + can be instantiated** (builds one and reports) — a cheap model smoke-test — + **reusing OOTB's probe instantiation code** (`verify_instantiation`, shared + with the base/plus probe). +- **× `minus`:** silly but allowed. `minus` says "instantiate nothing," yet + `--inspect-only` must build one scenario to verify — so **`--inspect-only` + takes priority**: a single verification instantiation happens, while the + *decision* stays minus-level (structural, no size profile fed into choices). +- **Suggestions:** only the config-time ones (nothing ran to yield + outcome-based ones). + +Ships in **PR1**: a plain `store_true` flag; the driver short-circuits after +printing, before apply-to-`Config` / run. + --- ## 6. Open details @@ -331,7 +357,8 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. together: fact-gathering (structural + one-scenario probe), the policy interpreter, EF-vs-cylinder / bundling / spoke selection, reporting (equivalent command line + post-run suggestions), apply-to-`Config`, the - `probe_scenarios` knob, quick-start docs, and tests. Ships the default + `probe_scenarios` knob, the `--inspect-only` dry run (§5.4, incl. the shared + `verify_instantiation`), quick-start docs, and tests. Ships the default on-ramp and the no-instantiation escape hatch in one PR. - **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds From f19a44f01b81a426a739ca8f299aa3eff0723464 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 13:49:57 -0700 Subject: [PATCH 09/34] Out-of-the-box: policy additional_options + computed suggestions Two refinements. (1) A policy additional_options list lets OOTB turn on extra command-line options beyond the structural choices (e.g. --max-iterations, or --grad-rho in a quality focus), applied via the user-options-win rule and skipped on the EF path; conditionality is expressed by which focus file ships the option, not by predicates. This subsumes param_defaults. (2) Suggestions are now mostly computed: small Python generators build messages from live facts/decision/run-outcome, so the prose lives in code and the policy only toggles/tunes them (suggestions.disabled). Decisions stay data-driven; suggestions are the computed diagnostics layer. Reworded the now-stale "named predicates" line in sec.5 to "plain coded checks". Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 26 +++++++++++++------ .../ootb_policies/ootb_policy_2026-06-28.json | 23 ++++++---------- 2 files changed, 26 insertions(+), 23 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 267307d33..338de69e7 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -108,7 +108,8 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): small enough per §5.1/§5.2 — otherwise the chosen cylinder configuration); **skipped entirely under `--inspect-only`** (§5.4). 4. A printed prioritized **Suggestions** list, emitted **after the run** - (labelled "Suggestions"; may reflect how the run went). + (labelled "Suggestions"). Mostly **computed** from live facts / decision / + run outcome, not canned (§5.1). --- @@ -141,8 +142,8 @@ Reasoning: *harder* to explain. - So: implement the "expert-system framing" (facts + declarative knowledge + thin matcher) as plain Python reading a JSON policy file (~order of 100 - lines). Conditions are **named predicates** the interpreter knows how to - evaluate — no expression language in v1. + lines). Conditions are **plain coded checks** over the facts — no expression + language in v1. **Migration path preserved:** when a benchmark corpus eventually exists it *tunes the numbers inside the next dated policy file* (e.g. a regression that @@ -190,11 +191,20 @@ data): | `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | | `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | | `bundle_sizing` | how big bundles are: largest `spb` within an effort budget (base = relative M; plus = measured seconds; minus = count); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | -| `param_defaults` | gap-fill only; minimal in v1 (`max_iterations`) | -| `suggestions` | named-predicate rules → the prioritized **Suggestions** list (req. 4), emitted after the run | - -**Predicates the interpreter must implement (v1):** `ran_ef_due_to_min_ranks`, -`chosen_solver_not_persistent`, `solver_is_lp_mip_only`, `cylinders_below_max`. +| `additional_options` | extra CLI options OOTB applies by default beyond the structural ones (e.g. `--max-iterations`; or `--grad-rho` in a quality focus); user options still win; skipped on the EF path | +| `suggestions` | toggles/tunes the **computed** suggestion generators (`disabled` suppresses specific ones); the prose lives in code, emitted after the run | + +**Additional options.** Beyond the structural choices, the policy +`additional_options` list applies extra flags (e.g. `--max-iterations`, or +`--grad-rho` in a quality focus) through the same user-options-win path; +**conditionality is expressed by *focus*** (which file ships the option), not by +predicates. These are decomposition-run options, skipped on the EF path. + +**Suggestions are computed, not canned.** The post-run **Suggestions** (req. 4, +§4) are produced by small Python *generators* that compute text from live +facts / decision / run-outcome; the policy only toggles/tunes them +(`suggestions.disabled`). This keeps the split clean: **decisions stay +data-driven (this policy); suggestions are the computed diagnostics layer.** ## 5.2 Effort tiers (instantiation depth) — RESOLVED 2026-06-28 diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index d128c8771..e0c7dc054 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -90,23 +90,16 @@ "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "fallback_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] }, - "param_defaults": { - "_comment": "Filled ONLY if the user did not set them. Kept minimal in v1 to avoid surprising the user; flag names not yet verified are intentionally omitted rather than guessed.", - "max_iterations": 100, - "_cold_start_guess": ["max_iterations"] + "additional_options": { + "_comment": "Extra command-line options OOTB turns on by default, BEYOND the structural choices (solver / EF / spokes / bundling). Applied via the same 'user options win' rule (requirement 0). CONDITIONALITY IS EXPRESSED BY FOCUS, not by predicates: a 'quality' focus file would list e.g. {\"flag\": \"--grad-rho\"} here while a 'quick' focus omits it. Each entry: flag, and value (null = a boolean flag). These are decomposition-run options (skipped on the EF path). Subsumes the old param_defaults. All cold-start.", + "options": [ + {"flag": "--max-iterations", "value": "100"} + ], + "_cold_start_guess": ["options"] }, "suggestions": { - "_comment": "Requirement 4: a prioritized 'Suggestions' list, printed AFTER the run executes (so it can also reflect how the run went). Each entry has a named predicate the interpreter evaluates (explainable; no expression language in v1) and a message; the run proceeds regardless.", - "rules": [ - {"id": "ran_ef_too_few_ranks", "when": "ran_ef_due_to_min_ranks", "priority": 1, - "message": "Ran the monolithic EF because fewer than 3 MPI ranks were available. With >= 3 ranks, mpi-sppy can run a parallel decomposition (hub + bound spokes) instead."}, - {"id": "no_persistent_solver", "when": "chosen_solver_not_persistent", "priority": 2, - "message": "The chosen solver has no persistent interface available. Installing/licensing a persistent interface (e.g. gurobi_persistent) lets subproblems warm-start and is typically much faster for PH."}, - {"id": "linearized_prox", "when": "solver_is_lp_mip_only", "priority": 3, - "message": "An LP/MIP-only solver was chosen, so the PH proximal term is being linearized (--linearize-proximal-terms). A QP-capable solver (gurobi/cplex/xpress, or ipopt for continuous models) avoids this approximation."}, - {"id": "more_ranks_help", "when": "cylinders_below_max", "priority": 4, - "message": "More MPI ranks would let OOTB add bound-tightening spokes (e.g. fwph, subgradient) and/or more intra-cylinder parallelism."} - ] + "_comment": "Requirement 4: a prioritized 'Suggestions' list, printed AFTER the run (so it can also reflect how the run went). The messages are MOSTLY COMPUTED in code (suggestion generators in mpisppy/generic/out_of_the_box.py): each inspects facts/decision/run-outcome and builds text with live values, so the PROSE lives in code, not here. Note the split from design sec. 5: decisions stay data-driven (this policy); suggestions are diagnostics and are computed. This block only TOGGLES/TUNES them -- 'disabled' lists generator names to suppress; thresholds (if any) can be added here for generators to read.", + "disabled": [] } } From 6948a23559b5eba1d1c7aeb3065cc8baa776ed86 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 14:32:03 -0700 Subject: [PATCH 10/34] Out-of-the-box: the minus tier cannot bundle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With no size profile there is no safe way to size a bundle, so minus always runs unbundled (bundling requires base or plus). Drops the count-based minus fallback: removes fallback_bundles_per_intra_rank from the policy and the _pick_spb_by_count helper from the sketch; recommend() now only sizes bundles when a probe profile is present. Reconciles §5.1/§5.2/§5.3/§6. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 17 ++++++++--------- .../ootb_policies/ootb_policy_2026-06-28.json | 5 ++--- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 338de69e7..ee88fae0c 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -190,7 +190,7 @@ data): | `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | | `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | | `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | -| `bundle_sizing` | how big bundles are: largest `spb` within an effort budget (base = relative M; plus = measured seconds; minus = count); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | +| `bundle_sizing` | how big bundles are (base/plus only — **minus cannot bundle**): largest `spb` within an effort budget (base = relative M; plus = measured seconds); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | | `additional_options` | extra CLI options OOTB applies by default beyond the structural ones (e.g. `--max-iterations`; or `--grad-rho` in a quality focus); user options still win; skipped on the EF path | | `suggestions` | toggles/tunes the **computed** suggestion generators (`disabled` suppresses specific ones); the prose lives in code, emitted after the run | @@ -218,7 +218,7 @@ to the tier. | Tier | Flag | Instantiates | New facts | What it can decide (vs. advise) | |---|---|---|---|---| -| minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability. Integrality/size unknown → only **advises** on prox linearization | +| minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability; **cannot bundle**. Integrality/size unknown → only **advises** on prox linearization | | base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int` | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; effort-budgeted bundle sizing (§5.3) | | plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | @@ -286,8 +286,8 @@ budget differs: at `plus_probe_solve_time_cap_seconds`), predict `t(spb) ≈ t₁·effort(spb)/effort(1)`, accept the largest `spb` with `t(spb) ≤ plus_target_seconds_per_bundle`. -- **minus** (no profile): effort is unknown → fall back to a conservative count - target (`fallback_bundles_per_intra_rank` bundles per intra-rank). +- **minus** (no profile): **cannot bundle** — with no model information there is + no safe way to size a bundle, so minus always runs unbundled. **Measurement does not remove the JSON assumptions.** `plus` only pins the *scale* (`t₁`); the *shape* (`int_exponent`, weights) still comes from @@ -307,8 +307,8 @@ EF gate stays **count/rank-based for now** (§5.1 `ef_fallback`) — reusing `_cold_start_guess`es; **foci** ship different shapes (a `mip-heavy` file with a steeper `int_exponent`), and the dated-file migration path (§5) refines the coefficients from benchmark data. The interpreter sketch implements the base -relative sizer (`_effort`, `_pick_spb_by_effort`) with `_pick_spb_by_count` as -the minus fallback; the `plus` measure-and-scale hook is stubbed. +relative sizer (`_effort`, `_pick_spb_by_effort`); minus does not bundle; the +`plus` measure-and-scale hook is stubbed. ## 5.4 `--inspect-only` (dry run; shares OOTB's instantiation) — RESOLVED 2026-06-28 @@ -343,9 +343,8 @@ printing, before apply-to-`Config` / run. base (one probe, default), plus (all + brief solve, later). - **Bundle sizing — RESOLVED** as an effort-budgeted rule (§5.3): policy `effort_scaling` shape + `bundle_sizing` budgets; interpreter `_effort` / - `_pick_spb_by_effort` (base, relative M) with `_pick_spb_by_count` (minus - fallback) and a stubbed `plus` measure-and-scale. Numbers are - `_cold_start_guess`es. + `_pick_spb_by_effort` (base, relative M); minus does not bundle; stubbed + `plus` measure-and-scale. Numbers are `_cold_start_guess`es. - **Still open:** an absolute, size-aware **EF gate** — reuse the effort *shape* with an absolute ceiling (unlike bundle sizing's relative budget) or defer to `plus`; count/rank-based for now (§5.3). diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index e0c7dc054..18a5fce69 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -79,15 +79,14 @@ }, "bundle_sizing": { - "_comment": "How big should proper bundles be (not just whether to bundle)? Proper bundling only (loose bundling removed in 2026); --scenarios-per-bundle MUST divide num_scens; library requires number_of_bundles >= number_of_ranks. Rule: pick the LARGEST scenarios_per_bundle that (a) divides num_scens, (b) leaves >= max(intra_ranks, min_bundles_per_intra_rank*intra_ranks) bundles, and (c) keeps a bundle's modeled effort within budget. base budget is RELATIVE: effort(spb)/effort(1) <= base_max_hardness_vs_single_scenario (unitless, portable, no measurement). plus budget is ABSOLUTE: measured_single_scenario_seconds * effort(spb)/effort(1) <= plus_target_seconds_per_bundle, with the measuring solve capped at plus_probe_solve_time_cap_seconds. Without a size profile (minus tier) effort is unknown, so fall back to fallback_bundles_per_intra_rank bundles per rank. Skip bundling below min_scens_to_consider_bundling. MIP solve times are noisy/non-monotone, so the shape in effort_scaling acts as a prior; a single measurement only sets the scale.", + "_comment": "How big should proper bundles be (not just whether to bundle)? Proper bundling only (loose bundling removed in 2026); --scenarios-per-bundle MUST divide num_scens; library requires number_of_bundles >= number_of_ranks. The MINUS tier CANNOT bundle: with no size profile there is no safe way to size a bundle, so minus always runs unbundled (bundling needs base or plus). For base/plus: pick the LARGEST scenarios_per_bundle that (a) divides num_scens, (b) leaves >= max(intra_ranks, min_bundles_per_intra_rank*intra_ranks) bundles, and (c) keeps a bundle's modeled effort within budget. base budget is RELATIVE: effort(spb)/effort(1) <= base_max_hardness_vs_single_scenario (unitless, portable, no measurement). plus budget is ABSOLUTE: measured_single_scenario_seconds * effort(spb)/effort(1) <= plus_target_seconds_per_bundle, with the measuring solve capped at plus_probe_solve_time_cap_seconds. Skip bundling below min_scens_to_consider_bundling. MIP solve times are noisy/non-monotone, so the shape in effort_scaling acts as a prior; a single measurement only sets the scale.", "min_scens_to_consider_bundling": 50, "min_bundles_per_intra_rank": 1, - "fallback_bundles_per_intra_rank": 2, "never_fewer_bundles_than_ranks": true, "base_max_hardness_vs_single_scenario": 10, "plus_target_seconds_per_bundle": 30, "plus_probe_solve_time_cap_seconds": 60, - "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "fallback_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] + "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] }, "additional_options": { From 53d22e70cb34765723b16e7b62da62616283efe5 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 14:37:25 -0700 Subject: [PATCH 11/34] Out-of-the-box: EF gate reuses the bundle effort model + an EF budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The EF is all scenarios as one model, so the EF gate now reuses effort() on the whole problem (effort(num_scens)) against an absolute EF budget, resolving the previously-open size-aware EF gate. Above the rank floor: base runs EF when effort(num_scens) <= ef_effort_budget; plus would use ef_target_seconds via a measured t1; minus (no profile) keeps the count rule ef_if_num_scens_at_most. The EF budget is absolute (the monolith has no single-scenario reference) but shares effort units with bundle sizing, so the two budgets are mutually consistent. Adds ef_effort_budget / ef_target_seconds to the policy; updates §5.1/§5.3/§6. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 25 +++++++++++-------- .../ootb_policies/ootb_policy_2026-06-28.json | 6 +++-- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index ee88fae0c..f97e71373 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -184,7 +184,7 @@ data): | Key | Purpose | |---|---| -| `ef_fallback` | `min_ranks_for_decomposition` (=3, req. 3) and `ef_if_num_scens_at_most` | +| `ef_fallback` | when to solve the EF: rank floor `min_ranks_for_decomposition` (req. 3); else the **same effort model as bundles** on the whole problem — base `effort(num_scens) ≤ ef_effort_budget`, plus measured `ef_target_seconds`, minus count `ef_if_num_scens_at_most` | | `solver` | `preference_order` (persistent-commercial → commercial → free QP-capable → LP/MIP-only), plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | | `hub` | default hub factory (`ph_hub`, no flag) | | `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | @@ -296,12 +296,17 @@ MIP can solve faster), so a single timing must not drive the whole choice — th JSON shape is a **prior/regularizer** that measurement calibrates. (Later refinement: `plus` measures two points to nudge `int_exponent` locally.) -**EF gate vs. bundle sizing.** Both use the same effort *shape*, but the EF gate -needs an **absolute** ceiling ("is the monolith small enough to solve as one -model?"); the relative-to-a-single-scenario trick does not gate the whole EF. -That absolute threshold is the genuinely hard part without measurement, so the -EF gate stays **count/rank-based for now** (§5.1 `ef_fallback`) — reusing -`effort` with an absolute ceiling (or deferring to `plus`) is a follow-on. +**EF gate uses the same effort model.** The EF is just all scenarios as one +model, so the EF gate reuses `effort()` on the whole problem: above the rank +floor, run the EF when `effort(num_scens)` is within an **EF budget**. Unlike +bundle sizing's *relative* budget (M× a single scenario), the EF budget is +**absolute** — the monolith has no single-scenario reference: **base** +`effort(num_scens) ≤ ef_effort_budget` (same effort units as bundle effort); +**plus** measured `t₁·effort(num_scens)/effort(1) ≤ ef_target_seconds`; **minus** +(no profile) falls back to the count rule `num_scens ≤ ef_if_num_scens_at_most`. +Because the units match bundle effort, the EF budget and bundle budgets are +mutually consistent: when the whole problem exceeds the EF budget, OOTB +decomposes and sizes each bundle within its own (smaller) effort budget. **Status.** All `effort_scaling` / `bundle_sizing` numbers are `_cold_start_guess`es; **foci** ship different shapes (a `mip-heavy` file with a @@ -345,9 +350,9 @@ printing, before apply-to-`Config` / run. `effort_scaling` shape + `bundle_sizing` budgets; interpreter `_effort` / `_pick_spb_by_effort` (base, relative M); minus does not bundle; stubbed `plus` measure-and-scale. Numbers are `_cold_start_guess`es. -- **Still open:** an absolute, size-aware **EF gate** — reuse the effort *shape* - with an absolute ceiling (unlike bundle sizing's relative budget) or defer to - `plus`; count/rank-based for now (§5.3). +- **EF gate — RESOLVED** (§5.3): reuses the bundle `effort()` model on the whole + problem against an absolute **EF budget** — base `ef_effort_budget`, plus + `ef_target_seconds`, minus the count rule `ef_if_num_scens_at_most`. - **Still open:** how the dated data files are generated, versioned, and shipped (the §5 migration path anticipates data-tuned successors). - **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 18a5fce69..441b585bf 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -6,10 +6,12 @@ "provenance": "COLD START. Authored by hand from mpi-sppy defaults and common practice; NO benchmark data yet. Every numeric threshold below is an educated guess flagged with _cold_start_guess and is expected to be replaced by data-tuned values in a later dated policy file.", "ef_fallback": { - "_comment": "Requirement 3: need at least min_ranks_for_decomposition ranks, else solve the EF. Also fall back to EF when there are too few scenarios for decomposition to buy anything.", + "_comment": "When to solve the monolithic EF instead of decomposing. Always EF when ranks < min_ranks_for_decomposition (requirement 3). Otherwise EF when the WHOLE problem is small enough, measured with the SAME effort model as bundle sizing (effort_scaling) applied to all scenarios as one model, i.e. effort(num_scens). Unlike bundle sizing's RELATIVE budget, the EF budget is ABSOLUTE (the monolith has no single-scenario reference). base: effort(num_scens) <= ef_effort_budget (same effort units as bundle effort). plus: measured_single_scenario_seconds * effort(num_scens)/effort(1) <= ef_target_seconds. minus (no size profile): fall back to the count rule num_scens <= ef_if_num_scens_at_most. All cold-start guesses.", "min_ranks_for_decomposition": 3, "ef_if_num_scens_at_most": 2, - "_cold_start_guess": ["ef_if_num_scens_at_most"] + "ef_effort_budget": 100000, + "ef_target_seconds": 120, + "_cold_start_guess": ["ef_if_num_scens_at_most", "ef_effort_budget", "ef_target_seconds"] }, "solver": { From e559f37473eece19f9aca44d76581af21d843fae Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 14:49:12 -0700 Subject: [PATCH 12/34] Out-of-the-box: per-concern option categories with superseded_by Splits the extra (non-structural) options into per-concern option_categories plus a catch-all additional_options. Each category block is {flag, value, superseded_by}, where superseded_by lists the user flags that obviate OOTB's default for that concern -- so OOTB backs off when the user addresses the concern with ANY equivalent flag, not just the identical one. This is required for correctness, not just tidiness: mpi-sppy allows only one rho setter active, so adding --grad-rho when the user chose --sensi-rho would be a hard error; rho_setter.superseded_by lists all rho setters. v1 categories: rho_setter (--grad-rho), termination (--rel-gap 0.01), max_iterations (--max-iterations 100), dynamic_rho (--dynamic-rho-primal-crit, boolean; threshold defaults 0.1). Catch-all entries may carry their own superseded_by (default = own flag). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 24 +++++++++++---- .../ootb_policies/ootb_policy_2026-06-28.json | 30 +++++++++++++++---- 2 files changed, 43 insertions(+), 11 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index f97e71373..2a2729dea 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -191,14 +191,26 @@ data): | `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | | `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | | `bundle_sizing` | how big bundles are (base/plus only — **minus cannot bundle**): largest `spb` within an effort budget (base = relative M; plus = measured seconds); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | -| `additional_options` | extra CLI options OOTB applies by default beyond the structural ones (e.g. `--max-iterations`; or `--grad-rho` in a quality focus); user options still win; skipped on the EF path | +| `option_categories` | per-concern default options (`rho_setter` `--grad-rho`, `termination` `--rel-gap 0.01`, `max_iterations` `--max-iterations 100`, `dynamic_rho` `--dynamic-rho-primal-crit`), each skipped if the user set any flag in its `superseded_by` list; skipped on the EF path | +| `additional_options` | catch-all for other extra flags (each with an optional `superseded_by`, default = its own flag) | | `suggestions` | toggles/tunes the **computed** suggestion generators (`disabled` suppresses specific ones); the prose lives in code, emitted after the run | -**Additional options.** Beyond the structural choices, the policy -`additional_options` list applies extra flags (e.g. `--max-iterations`, or -`--grad-rho` in a quality focus) through the same user-options-win path; -**conditionality is expressed by *focus*** (which file ships the option), not by -predicates. These are decomposition-run options, skipped on the EF path. +**Additional options, with per-concern override.** Beyond the structural +choices, the policy applies extra options grouped by **concern** in +`option_categories` — `rho_setter` (`--grad-rho`), `termination` +(`--rel-gap 0.01`), `max_iterations` (`--max-iterations 100`), `dynamic_rho` +(`--dynamic-rho-primal-crit`). Each carries a **`superseded_by`** list of user +flags that obviate OOTB's default for that concern, so OOTB backs off when the +user addresses the concern with *any* equivalent flag — not just the identical +one. This matters concretely: mpi-sppy allows **only one rho setter active**, so +adding `--grad-rho` when the user chose `--sensi-rho` would be a *hard error*, so +`rho_setter.superseded_by` lists all the rho setters. A leftover +`additional_options` catch-all handles miscellaneous flags (each with an optional +`superseded_by` that defaults to its own flag). Conditionality across problem +*classes* is still expressed by **focus** (which file ships which options). All +are decomposition-run options, skipped on the EF path. (`--dynamic-rho-primal-crit` +is a boolean — no argument; its threshold `dynamic_rho_primal_thresh` defaults to +0.1 — and needs an active rho setter, which OOTB's default `--grad-rho` provides.) **Suggestions are computed, not canned.** The post-run **Suggestions** (req. 4, §4) are produced by small Python *generators* that compute text from live diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 441b585bf..40a4336eb 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -91,12 +91,32 @@ "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] }, + "option_categories": { + "_comment": "Named option blocks OOTB applies by default, one per CONCERN, beyond the structural choices (solver / EF / spokes / bundling). Each is applied UNLESS the user already set any flag in its 'superseded_by' list -- the robust, per-concern form of 'user options win' (the user may address a concern with a DIFFERENT flag than OOTB's default). 'value' null = a boolean flag. These are decomposition-run options (skipped on the EF path). Foci may change them. All cold-start.", + "rho_setter": { + "flag": "--grad-rho", "value": null, + "superseded_by": ["--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho", "--reduced-costs-rho"], + "_note": "mpi-sppy enforces that only ONE rho setter is active, so OOTB adding --grad-rho when the user already picked another rho setter would be a HARD ERROR -- this is exactly why superseded_by lists all of them, not just --grad-rho." + }, + "termination": { + "flag": "--rel-gap", "value": "0.01", + "superseded_by": ["--rel-gap", "--abs-gap"] + }, + "max_iterations": { + "flag": "--max-iterations", "value": "100", + "superseded_by": ["--max-iterations"] + }, + "dynamic_rho": { + "flag": "--dynamic-rho-primal-crit", "value": null, + "superseded_by": ["--dynamic-rho-primal-crit", "--dynamic-rho-dual-crit"], + "_note": "Boolean flag (no argument needed); its paired threshold dynamic_rho_primal_thresh defaults to 0.1. Requires an active rho setter (config errors otherwise), which OOTB's default --grad-rho provides." + }, + "_cold_start_guess": ["rho_setter", "termination", "max_iterations", "dynamic_rho"] + }, + "additional_options": { - "_comment": "Extra command-line options OOTB turns on by default, BEYOND the structural choices (solver / EF / spokes / bundling). Applied via the same 'user options win' rule (requirement 0). CONDITIONALITY IS EXPRESSED BY FOCUS, not by predicates: a 'quality' focus file would list e.g. {\"flag\": \"--grad-rho\"} here while a 'quick' focus omits it. Each entry: flag, and value (null = a boolean flag). These are decomposition-run options (skipped on the EF path). Subsumes the old param_defaults. All cold-start.", - "options": [ - {"flag": "--max-iterations", "value": "100"} - ], - "_cold_start_guess": ["options"] + "_comment": "Catch-all for any OTHER extra flags not worth a named category. Each entry: flag, value (null = boolean), and optional superseded_by (defaults to [its own flag], i.e. plain per-flag override). For concern-level override (e.g. rho setters) use option_categories instead. Decomposition-run options (skipped on the EF path).", + "options": [] }, "suggestions": { From d84692e8489159788fef42e573e18a6f5d5be624 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 14:57:09 -0700 Subject: [PATCH 13/34] Out-of-the-box: user-forced decomposition overrides EF gate; validation TODO Two design items. (1) If the user explicitly set any decomposition flag (a wired spoke or non-default hub) and has at least the rank floor, OOTB never substitutes the EF, even for a small problem (requirement 0); the rank floor is still checked first. The decomposition-flag vocabulary is a fact about generic_cylinders, so it lives in the interpreter (DECOMPOSITION_FLAGS), not the policy. (2) Adds a Validation section (sec.8) sketching a fully automated policy-file validator that uses the examples: static schema checks plus behavioral checks (EF invoked when it should be, forced decomposition wins, bundling validity, no conflicting options, and round-trip executability of the emitted command line). Marked design TODO; checklist is partial. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 42 ++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 2a2729dea..451e5d9c9 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -320,6 +320,14 @@ Because the units match bundle effort, the EF budget and bundle budgets are mutually consistent: when the whole problem exceeds the EF budget, OOTB decomposes and sizes each bundle within its own (smaller) effort budget. +**User-forced decomposition overrides the gate.** If the user explicitly set any +**decomposition flag** — a wired spoke or a non-default hub (`DECOMPOSITION_FLAGS` +in the interpreter) — and has at least the rank floor, OOTB **never** substitutes +the EF, even for a small problem (requirement 0). The rank floor is checked +first: below it the decomposition can't fit, so the EF is used regardless. (This +flag vocabulary is a *fact* about `generic_cylinders`, not a focus preference, so +it lives in code, not the policy; the validator checks it against the real CLI.) + **Status.** All `effort_scaling` / `bundle_sizing` numbers are `_cold_start_guess`es; **foci** ship different shapes (a `mip-heavy` file with a steeper `int_exponent`), and the dated-file migration path (§5) refines the @@ -389,3 +397,37 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. - **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds iteration/time-limit defaults and solve-cost-aware bundling. + +--- + +## 8. Policy-file validation (TODO — tool to build) + +A **fully automated** validator that, given a policy file, checks it is correct +and produces sensible, *executable* configurations — using the mpi-sppy +**examples** as test models. Two layers: + +**Static (schema) checks.** JSON parses; required keys/types present; every +referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names +known, each `option_categories[*].flag` and every `superseded_by` entry is a +valid option, `DECOMPOSITION_FLAGS` match the `generic_cylinders` vocabulary; +`_cold_start_guess` entries name real keys; numbers in range. + +**Behavioral checks (using examples).** Run `recommend()` against real example +models (farmer, aircond, sizes, …) under synthetic environments (varying ranks, +available solvers, problem sizes) and assert: + +- **EF invoked when it should be:** small problem or `< min_ranks` ⇒ EF. +- **EF *not* invoked when it shouldn't be:** large / integer-heavy ⇒ decompose. +- **User-forced decomposition wins:** simulated user `--ph --lagrangian + --xhatshuffle` with ≥ rank floor ⇒ never EF (§5.3). +- **Bundling validity:** when bundling, `scenarios_per_bundle` divides + `num_scens` and `#bundles ≥ #ranks`. +- **No conflicting options:** `superseded_by` simulation ⇒ OOTB never stacks a + second rho setter (which would be a hard error). +- **Round-trip executability (strongest):** actually *run* OOTB's emitted + equivalent command line on the example (a short smoke run) and confirm it gets + past setup / completes — both for OOTB's own choice *and* for forced + decomposition, confirming decomposition "works more-or-less as expected." + +**Status: design TODO** — this checklist is partial; more checks to add. The +validator will gate the shipped policy files (eventually in CI). From a1e561a5d7952ba8d2d4b2150eab5f4cbe21a45b Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:07:18 -0700 Subject: [PATCH 14/34] Out-of-the-box: rank allocation -- small core, widened, unbalanced ratios MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three related changes. (1) max_cylinders 6 -> 7 so all six ladder rungs are reachable (no dead rung). (2) Roster sizing redesigned: prefer a small core (hub + lagrangian + xhatshuffle) widened by ranks, adding a further ladder rung only while each cylinder keeps >= min_ranks_per_cylinder ranks -- so 6 ranks gives 3 cylinders, not 6 single-rank cylinders. (3) Unbalanced rank distribution (flex-ranks): ranks split across chosen cylinders by per-cylinder rank_ratios (xhatter 0.2, default 1.0), normalized and floored at 1 each -- explicitly a crude cold-start, since the real split depends on subproblem solve cost (a plus-tier refinement). Adds §5.5; updates schema row and policy rank_allocation. The widest cylinder's rank count governs the bundling floor. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 26 ++++++++++++++++++- .../ootb_policies/ootb_policy_2026-06-28.json | 14 +++++++--- 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 451e5d9c9..10593e255 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -188,7 +188,7 @@ data): | `solver` | `preference_order` (persistent-commercial → commercial → free QP-capable → LP/MIP-only), plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | | `hub` | default hub factory (`ph_hub`, no flag) | | `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | -| `rank_allocation` | fill the ladder first, then spend leftover ranks on intra-cylinder parallelism | +| `rank_allocation` | small-core roster widened by ranks (add a rung only while each cylinder keeps ≥ `min_ranks_per_cylinder`, ≤ `max_cylinders`); ranks split **unbalanced** across cylinders by `rank_ratios` (xhatter 0.2) — crude cold-start (§5.5) | | `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | | `bundle_sizing` | how big bundles are (base/plus only — **minus cannot bundle**): largest `spb` within an effort budget (base = relative M; plus = measured seconds); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | | `option_categories` | per-concern default options (`rho_setter` `--grad-rho`, `termination` `--rel-gap 0.01`, `max_iterations` `--max-iterations 100`, `dynamic_rho` `--dynamic-rho-primal-crit`), each skipped if the user set any flag in its `superseded_by` list; skipped on the EF path | @@ -360,6 +360,30 @@ config-time suggestions, then stop before the production optimization run.** Ships in **PR1**: a plain `store_true` flag; the driver short-circuits after printing, before apply-to-`Config` / run. +## 5.5 Rank allocation — small core, widened, and unbalanced + +Two parts (policy `rank_allocation`): + +**Roster size — prefer width over weak spokes.** Start from the minimal core +(≥1 outer + ≥1 inner spoke = hub + `--lagrangian` + `--xhatshuffle`). Add +further ladder rungs only while each cylinder would still keep +`min_ranks_per_cylinder` ranks (a coarse uniform gate), up to `max_cylinders` +(now **7**, so all six ladder rungs are reachable at enough ranks — no dead +rung). So **6 ranks → hub + lagrangian + xhatshuffle (3 cylinders), widened** — +*not* 6 single-rank cylinders. Extra ranks buy subproblem throughput, which (so +far in practice) beats piling on lower-value bound spokes. + +**Unbalanced distribution (flex-ranks).** Ranks are split across the chosen +cylinders by per-cylinder **`rank_ratios`**, *not* uniformly: cheaper cylinders +get a smaller share. v1 ships `--xhatshuffle` (and the xhat family) at **0.2**; +everything else uses `default_rank_ratio` 1.0. Ratios are normalized over the +chosen cylinders and floored at 1 rank each (e.g. 6 ranks → hub 3, lagrangian 2, +xhatshuffle 1). **This is a crude cold-start:** the right split is a much more +complicated calculation that depends on the *nature of the subproblems* +(relative solve cost), and is a natural place for the `plus` tier's measurements +to inform. The widest cylinder's rank count governs the bundling +`#bundles ≥ #ranks` floor. + --- ## 6. Open details diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 40a4336eb..692f59bc3 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -61,14 +61,20 @@ "outer": 1, "inner": 1 }, - "max_cylinders": 6, + "max_cylinders": 7, "_cold_start_guess": ["max_cylinders", "ladder ordering of priorities 3-6"] }, "rank_allocation": { - "_comment": "How to spend ranks. First fill the spoke ladder up to max_cylinders (each cylinder = 1 rung + the hub). Then, rather than adding lower-value spokes, spend remaining ranks on intra-cylinder parallelism (more ranks per cylinder => faster subproblem throughput), especially when there are many scenarios/bundles. v1 keeps this simple and conservative.", - "fill_ladder_before_intra_cylinder": true, - "extra_ranks_policy": "intra_cylinder" + "_comment": "Two parts. (A) ROSTER SIZE: prefer a SMALL CORE (>=1 outer + >=1 inner spoke) widened by ranks over a wide roster of weak spokes. Start from the core; add further ladder rungs only while each cylinder would still get >= min_ranks_per_cylinder ranks (uniform/nominal basis -- a coarse gate) and we stay <= max_cylinders. So 6 ranks -> hub + lagrangian + xhatshuffle (3 cylinders, widened), NOT 6 single-rank cylinders. (B) UNBALANCED DISTRIBUTION (flex-ranks): split ranks across the chosen cylinders by per-cylinder RATIOS, not uniformly -- cheaper cylinders get fewer ranks. ratios are normalized over the chosen cylinders and floored at 1 rank each. WARNING: rank_ratios here are a CRUDE cold-start (e.g. xhatter 0.2); the right split is a much more complicated calculation that depends on the NATURE OF THE SUBPROBLEMS (relative solve cost), ideally informed by the plus tier's measurements.", + "min_ranks_per_cylinder": 2, + "default_rank_ratio": 1.0, + "rank_ratios": { + "--xhatshuffle": 0.2, + "--xhatxbar": 0.2, + "--xhatlshaped": 0.2 + }, + "_cold_start_guess": ["min_ranks_per_cylinder", "default_rank_ratio", "rank_ratios"] }, "effort_scaling": { From e1ba4bc0d969bd8ffb1afe3ba7ebe2b09b345cd5 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:10:37 -0700 Subject: [PATCH 15/34] Out-of-the-box: --inspect-only takes an optional assumed rank count --inspect-only N plans as if N ranks were available, so a supercomputer user can get the recommended command line for a large job from a login node without launching it; bare --inspect-only uses the detected ranks. Only the rank count is hypothetical -- solvers and model size still come from the real session -- and the emitted "mpiexec -np N ..." reflects it. This makes --inspect-only an optional-value flag (domain=str, nargs='?'), not store_true (same bool-domain caveat as the effort flags). Updates sec.5.4. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 10593e255..54353d9c7 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -356,9 +356,17 @@ config-time suggestions, then stop before the production optimization run.** *decision* stays minus-level (structural, no size profile fed into choices). - **Suggestions:** only the config-time ones (nothing ran to yield outcome-based ones). - -Ships in **PR1**: a plain `store_true` flag; the driver short-circuits after -printing, before apply-to-`Config` / run. +- **Optional assumed rank count (HPC planning):** `--inspect-only N` plans as if + `N` ranks were available — so a supercomputer user can get the recommended + command line for, say, a 512-rank job *from a login node, without launching + it*. Bare `--inspect-only` uses the actually-detected ranks. Everything else + (solvers, model size) still comes from the real (possibly small) session; only + the rank count is hypothetical, and the emitted `mpiexec -np N …` reflects it. + +Ships in **PR1**: an **optional-value** flag (`domain=str, nargs='?'`, the value +being the assumed rank count) — *not* `store_true`, since it now takes a value +(the same bool-domain caveat as the effort flags). The driver short-circuits +after printing, before apply-to-`Config` / run. ## 5.5 Rank allocation — small core, widened, and unbalanced From 07a6effc972ea657be07213ac9bd6bb12ebb81f9 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:18:57 -0700 Subject: [PATCH 16/34] Out-of-the-box: validator runs decomposition + produces a report Expands the validation design (sec.8) into three layers fast-to-slow: static schema checks, fast decision checks (recommend() only), and a slower run tier that actually executes on the small examples. The run tier forces cylinders even where the EF gate would pick EF, runs them to near convergence, and verifies the behavior is about what we expect by comparing to the EF ground truth (objective/first-stage within tolerance, bounds don't cross) -- feasible because the small examples are EF-solvable. These need longer runs, so the run tier is nightly/opt-in while the fast tiers gate every change. Adds a report requirement: pass/fail per check with details, machine- and human-readable, naming the policy file and version. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 47 +++++++++++++++++++++------- 1 file changed, 35 insertions(+), 12 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 54353d9c7..bed70014c 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -435,18 +435,19 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. ## 8. Policy-file validation (TODO — tool to build) A **fully automated** validator that, given a policy file, checks it is correct -and produces sensible, *executable* configurations — using the mpi-sppy -**examples** as test models. Two layers: +and produces configurations that are *executable* and *behave about as we +expect* — using the mpi-sppy **examples** as test models. Three layers, fast to +slow: -**Static (schema) checks.** JSON parses; required keys/types present; every +**1. Static (schema) checks.** JSON parses; required keys/types present; every referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names known, each `option_categories[*].flag` and every `superseded_by` entry is a valid option, `DECOMPOSITION_FLAGS` match the `generic_cylinders` vocabulary; `_cold_start_guess` entries name real keys; numbers in range. -**Behavioral checks (using examples).** Run `recommend()` against real example -models (farmer, aircond, sizes, …) under synthetic environments (varying ranks, -available solvers, problem sizes) and assert: +**2. Decision checks (fast; `recommend()` only, no solves).** Run `recommend()` +against real example models (farmer, aircond, sizes, …) under synthetic +environments (varying ranks, solvers, problem sizes) and assert: - **EF invoked when it should be:** small problem or `< min_ranks` ⇒ EF. - **EF *not* invoked when it shouldn't be:** large / integer-heavy ⇒ decompose. @@ -456,10 +457,32 @@ available solvers, problem sizes) and assert: `num_scens` and `#bundles ≥ #ranks`. - **No conflicting options:** `superseded_by` simulation ⇒ OOTB never stacks a second rho setter (which would be a hard error). -- **Round-trip executability (strongest):** actually *run* OOTB's emitted - equivalent command line on the example (a short smoke run) and confirm it gets - past setup / completes — both for OOTB's own choice *and* for forced - decomposition, confirming decomposition "works more-or-less as expected." -**Status: design TODO** — this checklist is partial; more checks to add. The -validator will gate the shipped policy files (eventually in CI). +**3. Run checks (slow; actually execute — *longer runs*).** The decision checks +only verify *what OOTB chooses*; this layer verifies it *works and behaves about +right* by really running on the (small) examples: + +- **Round-trip executability:** run OOTB's emitted command line and confirm it + completes (not just gets past setup). +- **Forced decomposition on small problems:** force cylinders **even where the EF + gate would have picked the EF**, run them to (near) convergence, and verify the + decomposition behaves as expected — it converges, bounds don't cross, and the + result **matches the EF ground truth** (objective / first-stage solution within + tolerance). Small examples make this possible: the EF gives the true optimum to + compare against. (So the validator runs *both* the EF and the forced + decomposition on the same small model and compares.) +- Spot-check a few rank counts / bundlings so decomposition runs more-or-less as + expected across the configurations OOTB would actually produce. + +These runs are **not cheap** — PH to convergence takes time — so layer 3 is a +**heavier, slower tier** (nightly / opt-in), separate from the fast static + +decision checks that can gate every change. + +**Report.** The validator produces a **report**: pass/fail per check with +details — which example and synthetic environment, expected vs. actual, and for +layer-3 runs the EF-vs-decomposition objective gap and convergence status. It is +both **machine-readable** (so CI can gate on it) and **human-readable** (a +summary), and it names the policy file and `policy_version` validated. + +**Status: design TODO** — checklist still partial; more to add. The validator +will gate the shipped policy files (fast tiers in CI; the run tier nightly). From 03c7cc7255b133dae9166f4b9dce7c0de8ddab97 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:21:38 -0700 Subject: [PATCH 17/34] Out-of-the-box: only a small, solver-free part of the validator gates CI CI runners typically have no commercial solver and tight time budgets, so the per-PR gate is limited to the cheap, solver-free checks: layer 1 (static schema) plus the synthetic-facts subset of layer 2 (pure recommend() on hand-built Facts, no instantiation or solver). Example instantiation, anything needing a solver, and all of layer 3 (real runs) execute nightly / on demand / locally, not as a gate. Corrects the earlier "fast tiers in CI" overstatement in sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index bed70014c..4d1e95f17 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -474,9 +474,9 @@ right* by really running on the (small) examples: - Spot-check a few rank counts / bundlings so decomposition runs more-or-less as expected across the configurations OOTB would actually produce. -These runs are **not cheap** — PH to convergence takes time — so layer 3 is a -**heavier, slower tier** (nightly / opt-in), separate from the fast static + -decision checks that can gate every change. +These runs are **not cheap** — PH to convergence takes time, and they need a +real solver — so layer 3 runs **nightly / on demand / locally**, never as a +per-PR gate. **Report.** The validator produces a **report**: pass/fail per check with details — which example and synthetic environment, expected vs. actual, and for @@ -484,5 +484,11 @@ layer-3 runs the EF-vs-decomposition objective gap and convergence status. It is both **machine-readable** (so CI can gate on it) and **human-readable** (a summary), and it names the policy file and `policy_version` validated. -**Status: design TODO** — checklist still partial; more to add. The validator -will gate the shipped policy files (fast tiers in CI; the run tier nightly). +**Only a small part can gate CI.** CI runners typically have **no commercial +solver** and tight time budgets, so the per-PR gate is limited to the cheap, +solver-free checks: layer 1 (static schema) plus the **synthetic-facts** subset +of layer 2 (pure `recommend()` decisions on hand-built `Facts`, no instantiation, +no solver). Everything else — example instantiation, anything needing a solver, +and all of layer 3 — runs **nightly / on demand / locally**, not as a gate. + +**Status: design TODO** — checklist still partial; more to add. From b5b49a2793d6de859776e480c98073fd9f79917f Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:22:49 -0700 Subject: [PATCH 18/34] Out-of-the-box: validator report details all tests, flags maxed-out iterations The report details every test (not just failures) and, for layer-3 runs, records the iteration count and convergence status. It prominently highlights runs that hit the iteration cap without converging ("maxed out on iterations") -- a yellow flag that the recommended config or the policy's defaults converge poorly on that example, even when the run technically completed. Updates sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 4d1e95f17..5f090eace 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -478,11 +478,15 @@ These runs are **not cheap** — PH to convergence takes time, and they need a real solver — so layer 3 runs **nightly / on demand / locally**, never as a per-PR gate. -**Report.** The validator produces a **report**: pass/fail per check with -details — which example and synthetic environment, expected vs. actual, and for -layer-3 runs the EF-vs-decomposition objective gap and convergence status. It is -both **machine-readable** (so CI can gate on it) and **human-readable** (a -summary), and it names the policy file and `policy_version` validated. +**Report.** The validator produces a **report** that details **every** test, not +just failures: which example and synthetic environment, expected vs. actual, and +for layer-3 runs the EF-vs-decomposition objective gap, **iteration count**, and +convergence status. It **prominently highlights runs that hit the iteration cap +without converging** ("maxed out on iterations") — a yellow flag that the +recommended configuration (or the policy's defaults) converges poorly on that +example, *even when the run technically completed*. The report is both +**machine-readable** (CI gating) and **human-readable** (a summary), and it names +the policy file and `policy_version` validated. **Only a small part can gate CI.** CI runners typically have **no commercial solver** and tight time budgets, so the per-PR gate is limited to the cheap, From 879e0a2466d4dd86a25b1f7d7514345af54d378f Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:26:59 -0700 Subject: [PATCH 19/34] Out-of-the-box: validator is a PR1 deliverable (not TBD); nail down its shape The validator is no longer a vague TODO: a dated policy file should not ship without the tool that validates it, so it lands in PR1. Nails down the concrete shape -- a runnable mpisppy/generic/ootb_validate.py over a fixed small example set (farmer, a small MIP, aircond), with the CI gate a pytest of the static + synthetic-facts layers wired into run_coverage.bash and test_pr_and_main.yml, and layer 3 run via --run nightly/locally. Adds the validator to PR1 scope in sec.7 and reworded the heading to "validator" (the tool), not "validation". Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 29 ++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 5f090eace..261f367ba 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -424,20 +424,22 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. interpreter, EF-vs-cylinder / bundling / spoke selection, reporting (equivalent command line + post-run suggestions), apply-to-`Config`, the `probe_scenarios` knob, the `--inspect-only` dry run (§5.4, incl. the shared - `verify_instantiation`), quick-start docs, and tests. Ships the default - on-ramp and the no-instantiation escape hatch in one PR. + `verify_instantiation`), the **policy-file validator (§8)** with its CI-gating + layers wired into CI, quick-start docs, and tests. Ships the default on-ramp + and the no-instantiation escape hatch in one PR. - **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds iteration/time-limit defaults and solve-cost-aware bundling. --- -## 8. Policy-file validation (TODO — tool to build) +## 8. Policy-file validator (PR1 deliverable) A **fully automated** validator that, given a policy file, checks it is correct and produces configurations that are *executable* and *behave about as we -expect* — using the mpi-sppy **examples** as test models. Three layers, fast to -slow: +expect* — using the mpi-sppy **examples** as test models. **It ships with PR1:** a +dated policy file should not be shipped without the tool that validates it. Three +layers, fast to slow: **1. Static (schema) checks.** JSON parses; required keys/types present; every referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names @@ -495,4 +497,19 @@ of layer 2 (pure `recommend()` decisions on hand-built `Facts`, no instantiation no solver). Everything else — example instantiation, anything needing a solver, and all of layer 3 — runs **nightly / on demand / locally**, not as a gate. -**Status: design TODO** — checklist still partial; more to add. +**Concrete shape (built in PR1).** A runnable module +`mpisppy/generic/ootb_validate.py` — `python -m mpisppy.generic.ootb_validate +` — exercising a fixed small example set: **farmer** (two-stage, +continuous), a small **MIP** example (e.g. `sizes`/`sslp`), and **aircond** +(multistage). Decision checks sweep a handful of synthetic +`(ranks, solvers, num_scens, size/integrality)` tuples chosen to hit each branch +(EF-small, EF-few-ranks, decompose-large, forced-decomp, bundling on/off, LP-only +solver, no-persistent). The CI gate is a **pytest** that runs only layers 1 + +2-synthetic on the shipped policy file(s) — and, per project convention, is wired +into `run_coverage.bash` **and** `test_pr_and_main.yml` in the same commit. Layer +3 is the same module invoked with `--run` (nightly / local). The report is +written human-readable and machine-readable (JSON). + +**Status:** design nailed down; **scheduled for PR1**. The check *catalog* will +keep growing, but the tool, the three-layer structure, the CI gate, and the +report all ship in PR1. From 821bad10e809ff7cd01bf6abbdc982e402c61736 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:29:11 -0700 Subject: [PATCH 20/34] Out-of-the-box: full validator (incl. run-tier) in PR1 -- it's the try-it path The validator is also the easiest way to try OOTB out: running it on the bundled examples exercises the whole pipeline end-to-end (decisions, real decomposition runs, EF comparison, the report), so it doubles as the demonstration harness. That means the full tool -- all three layers, including the run-tier -- ships in PR1, not a fast-follow; the small example set keeps the run-tier feasible with a modest/free solver. Records the rationale in sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 261f367ba..1caaa8b06 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -437,9 +437,15 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. A **fully automated** validator that, given a policy file, checks it is correct and produces configurations that are *executable* and *behave about as we -expect* — using the mpi-sppy **examples** as test models. **It ships with PR1:** a -dated policy file should not be shipped without the tool that validates it. Three -layers, fast to slow: +expect* — using the mpi-sppy **examples** as test models. **It ships with PR1** +for two reasons: (1) a dated policy file should not be shipped without the tool +that validates it; and (2) it is the **easiest way to *try OOTB out*** — running +it on the bundled examples exercises the whole pipeline end-to-end (decisions, +real decomposition runs, EF comparison, the report), so it doubles as the +demonstration harness. Because of (2), the **full tool — all three layers, +including the run-tier — is in PR1** (not a fast-follow); the small example set +keeps the run-tier feasible with a modest/free solver. Three layers, fast to +slow: **1. Static (schema) checks.** JSON parses; required keys/types present; every referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names From 411e27832c91993d81963c76b3d32a4295bb2cc6 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:39:48 -0700 Subject: [PATCH 21/34] Out-of-the-box: internal-consistency pass (and reconcile with the policy JSON) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audited the design doc against itself and the committed policy file. The schema table is an exact set match with the JSON config keys. Fixed four internal drifts: the status header now lists all resolved sections (not just §5/§5.2); sec.4 cites the EF gate at §5.3 (was §5.2); dropped the stale "~100 lines" interpreter-size claim; and sec.8 layer 2 now introduces its synthetic-facts (CI-gating) vs real-example (out-of-CI) split so the later CI note has a clear referent. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 1caaa8b06..1ee35a223 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -1,10 +1,12 @@ # Out-of-the-box auto-configuration — design **Status:** Design phase. Branch `outOfTheBox` (off Pyomo/mpi-sppy `main`), on -the DLWoodruff fork. Proceeding deliberately ("slowly"): requirements -confirmed; the decision-logic mechanism (§5) and instantiation effort tiers -(§5.2) are resolved; the first dated policy file is committed. No production -library code yet — only an uncommitted interpreter *sketch* (§7). +the DLWoodruff fork. Proceeding deliberately ("slowly"): requirements confirmed +and the major design questions resolved — decision-logic mechanism (§5), policy +file + path selection (§5.1), effort tiers (§5.2), bundle sizing & the EF gate +(§5.3), `--inspect-only` (§5.4), rank allocation (§5.5), and the PR1 validator +(§8). The first dated policy file is committed. No production library code yet — +only an uncommitted interpreter *sketch* (§7). **Author:** dlw (captured with Claude Code assistance) **Last updated:** 2026-06-28 @@ -105,7 +107,7 @@ These were raised as scoping questions and confirmed by the user (2026-06-28): executes. 2. A printed **equivalent explicit command line** (up front, before the run). 3. The run itself (EF when the EF gate trips — too few ranks, or the problem is - small enough per §5.1/§5.2 — otherwise the chosen cylinder configuration); + small enough per §5.1/§5.3 — otherwise the chosen cylinder configuration); **skipped entirely under `--inspect-only`** (§5.4). 4. A printed prioritized **Suggestions** list, emitted **after the run** (labelled "Suggestions"). Mostly **computed** from live facts / decision / @@ -141,8 +143,8 @@ Reasoning: dependency into already-fragile MPI/solver installs and can make firing-order *harder* to explain. - So: implement the "expert-system framing" (facts + declarative knowledge + - thin matcher) as plain Python reading a JSON policy file (~order of 100 - lines). Conditions are **plain coded checks** over the facts — no expression + thin matcher) as a thin, hand-written Python interpreter reading a JSON policy + file. Conditions are **plain coded checks** over the facts — no expression language in v1. **Migration path preserved:** when a benchmark corpus eventually exists it @@ -454,8 +456,10 @@ valid option, `DECOMPOSITION_FLAGS` match the `generic_cylinders` vocabulary; `_cold_start_guess` entries name real keys; numbers in range. **2. Decision checks (fast; `recommend()` only, no solves).** Run `recommend()` -against real example models (farmer, aircond, sizes, …) under synthetic -environments (varying ranks, solvers, problem sizes) and assert: +on **hand-built synthetic `Facts`** (no instantiation — the CI-gating subset) +and, more thoroughly, on **real example models** (farmer, aircond, sizes, …; +probe-instantiated, hence out of CI) under synthetic environments (varying ranks, +solvers, problem sizes), and assert: - **EF invoked when it should be:** small problem or `< min_ranks` ⇒ EF. - **EF *not* invoked when it shouldn't be:** large / integer-heavy ⇒ decompose. From 585bfa8f19f83c602c9547a0e524690cd53692b0 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:47:28 -0700 Subject: [PATCH 22/34] Out-of-the-box: validator flags problems, does not assert expected results The validator is not a correctness oracle -- we can't reasonably include expected results. Instead it runs the recommended configs, records what happened, and automatically flags two failure modes: EF recommended but not at a 1% MIP gap within ten minutes, and decomposition that terminates on max_iterations (including forced cylinders on small problems). Everything else is recorded for a human reader, who judges whether performance is acceptable -- which is why the report must be human-readable. Drops the EF-ground-truth / expected-vs-actual framing from sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 75 ++++++++++++++-------------- 1 file changed, 38 insertions(+), 37 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 1ee35a223..f3840d663 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -437,17 +437,19 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. ## 8. Policy-file validator (PR1 deliverable) -A **fully automated** validator that, given a policy file, checks it is correct -and produces configurations that are *executable* and *behave about as we -expect* — using the mpi-sppy **examples** as test models. **It ships with PR1** -for two reasons: (1) a dated policy file should not be shipped without the tool -that validates it; and (2) it is the **easiest way to *try OOTB out*** — running -it on the bundled examples exercises the whole pipeline end-to-end (decisions, -real decomposition runs, EF comparison, the report), so it doubles as the -demonstration harness. Because of (2), the **full tool — all three layers, -including the run-tier — is in PR1** (not a fast-follow); the small example set -keeps the run-tier feasible with a modest/free solver. Three layers, fast to -slow: +A **fully automated** validator that, given a policy file, checks it is +well-formed and that its recommended configurations **run** — and **flags the +runs that look problematic** for a human to review — using the mpi-sppy +**examples** as test models. It does **not** assert correctness or "expected +results" (we have no cheap, reliable oracle for that); the reader judges whether +performance is acceptable, which is why the report is human-readable. **It ships +with PR1** for two reasons: (1) a dated policy file should not be shipped without +the tool that validates it; and (2) it is the **easiest way to *try OOTB out*** +— running it on the bundled examples exercises the whole pipeline end-to-end +(decisions, real runs, the report), so it doubles as the demonstration harness. +Because of (2), the **full tool — all three layers, including the run-tier — is +in PR1** (not a fast-follow); the small example set keeps the run-tier feasible +with a modest/free solver. Three layers, fast to slow: **1. Static (schema) checks.** JSON parses; required keys/types present; every referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names @@ -471,34 +473,33 @@ solvers, problem sizes), and assert: second rho setter (which would be a hard error). **3. Run checks (slow; actually execute — *longer runs*).** The decision checks -only verify *what OOTB chooses*; this layer verifies it *works and behaves about -right* by really running on the (small) examples: - -- **Round-trip executability:** run OOTB's emitted command line and confirm it - completes (not just gets past setup). -- **Forced decomposition on small problems:** force cylinders **even where the EF - gate would have picked the EF**, run them to (near) convergence, and verify the - decomposition behaves as expected — it converges, bounds don't cross, and the - result **matches the EF ground truth** (objective / first-stage solution within - tolerance). Small examples make this possible: the EF gives the true optimum to - compare against. (So the validator runs *both* the EF and the forced - decomposition on the same small model and compares.) -- Spot-check a few rank counts / bundlings so decomposition runs more-or-less as - expected across the configurations OOTB would actually produce. - -These runs are **not cheap** — PH to convergence takes time, and they need a -real solver — so layer 3 runs **nightly / on demand / locally**, never as a -per-PR gate. +only verify *what OOTB chooses*; this layer actually **runs** the recommended +configurations on the (small) examples and records what happened. We can't assert +"expected results," but two failure modes are worth **flagging automatically**: + +- **EF recommended but slow:** when OOTB recommends the EF, run it and **flag any + that fail to reach a 1% MIP gap within ten minutes** — a sign the EF was the + wrong call for that problem. +- **Cylinders maxed out on iterations:** run the decomposition — including + **forced** cylinders where the EF gate would otherwise have picked the EF, to + exercise that path — and **flag any that terminate on `max_iterations`** rather + than converging. + +Everything else (objective, bound/gap, iteration count, wall time per run) is +**recorded for the reader**, not auto-judged. The 1%-gap and ten-minute +thresholds are validator settings, not policy. These runs are **not cheap** — PH +to convergence takes time, and they need a real solver — so layer 3 runs +**nightly / on demand / locally**, never as a per-PR gate. **Report.** The validator produces a **report** that details **every** test, not -just failures: which example and synthetic environment, expected vs. actual, and -for layer-3 runs the EF-vs-decomposition objective gap, **iteration count**, and -convergence status. It **prominently highlights runs that hit the iteration cap -without converging** ("maxed out on iterations") — a yellow flag that the -recommended configuration (or the policy's defaults) converges poorly on that -example, *even when the run technically completed*. The report is both -**machine-readable** (CI gating) and **human-readable** (a summary), and it names -the policy file and `policy_version` validated. +just failures: which example and synthetic environment, and for layer-3 runs the +objective, bound/gap, **iteration count**, and wall time. It does **not** print +"expected vs. actual" — there is no oracle — so the reader verifies that +performance is acceptable, which is exactly why the report must be +**human-readable** (a summary) as well as **machine-readable**. It **prominently +highlights** the two automatic flags: EF runs that **missed a 1% gap in ten +minutes**, and decomposition runs that **maxed out on iterations**. It names the +policy file and `policy_version` validated. **Only a small part can gate CI.** CI runners typically have **no commercial solver** and tight time budgets, so the per-PR gate is limited to the cheap, From a655ae71872f59fad296d859c028815fa2b1d201 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:50:04 -0700 Subject: [PATCH 23/34] Out-of-the-box: add effort-calibration tool (sec.9, future) A third tool (after the interpreter and validator), data-tuning side and not PR1: it fits the effort_scaling coefficients from timed solves across a benchmark spread so modeled effort tracks measured time, and produces an effort->seconds scale so the abstract budgets (ef_effort_budget, the base M) can be set and read in seconds (matching the existing plus_*_seconds budgets). Its output is updated coefficients + scale for a new dated policy file -- the producer side of the sec.5 migration path. sec.6 now names it as the dated-file generation mechanism (corpus/versioning cadence still open). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 35 ++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index f3840d663..2217c0176 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -407,8 +407,9 @@ to inform. The widest cylinder's rank count governs the bundling - **EF gate — RESOLVED** (§5.3): reuses the bundle `effort()` model on the whole problem against an absolute **EF budget** — base `ef_effort_budget`, plus `ef_target_seconds`, minus the count rule `ef_if_num_scens_at_most`. -- **Still open:** how the dated data files are generated, versioned, and shipped - (the §5 migration path anticipates data-tuned successors). +- **Dated data files — generation mechanism is the effort-calibration tool + (§9)** (fits coefficients + an effort→seconds scale from a benchmark corpus); + the corpus, versioning, and shipping cadence are **still open**. - **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. --- @@ -524,3 +525,33 @@ written human-readable and machine-readable (JSON). **Status:** design nailed down; **scheduled for PR1**. The check *catalog* will keep growing, but the tool, the three-layer structure, the CI gate, and the report all ship in PR1. + +--- + +## 9. Effort-calibration tool (future; data-tuning side) + +The `effort_scaling` coefficients and the effort budgets ship as abstract +cold-start guesses (§5.3) in arbitrary "effort units." A separate **calibration +tool** turns them into data-tuned, *interpretable* values: + +- **Fit the coefficients.** Run timed solves across a benchmark spread (varying + continuous / integer var counts, scenarios-per-bundle, nonant counts) and fit + `cont_coeff`, `int_weight`, `int_exponent`, `int_nonant_coeff` so that modeled + `effort(...)` tracks measured solve time. +- **Translate effort ↔ seconds.** Produce an effort→time scale so the abstract + budgets (`ef_effort_budget`, the base `M`, …) can be set and read in + **seconds** — the unit the `plus_*` budgets already use (minutes = seconds/60 + for humans) — e.g. "run the EF only if it should finish in ~600 s" instead of + opaque effort units. +- **Output.** Updated coefficients + the effort→seconds scale for a new **dated** + policy file — the producer side of the §5 "dated data files refined by + benchmark data" migration path. + +Caveats: solve time is machine- and solver-dependent and (for MIPs) noisy, so a +calibrated scale is per reference machine/solver and approximate. The **plus** +tier effectively *re-calibrates per session* via its measured single-scenario +time (`t₁`); **base** relies on this offline calibration. + +**Status:** future tool (not PR1). The on-ramp ships with cold-start +coefficients; calibration produces the first data-tuned dated policy once a +benchmark corpus exists. From 073a7f0e3807c79c6d6ae31d2d2b9ec79cfc81ce Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:55:51 -0700 Subject: [PATCH 24/34] Out-of-the-box: calibration tool moves into PR1 (policy numbers are its output) A policy whose effort coefficients and budgets are hand-guesses is unassessable -- reviewers would be reviewing noise. So the effort-calibration tool ships in PR1, and the v1 policy's effort numbers are its output: an initial calibration on the example set (sharing the validator run-tier's timed solves), reproducible by re-running it; a broader benchmark corpus is the future refinement. Updates sec.9 status, the PR1 scope (sec.7), the schema note (sec.5.1), sec.6, and the status header. PR1 is now interpreter + validator + calibrator -- large but cohesive: together they make the shipped policy runnable and assessable. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 42 ++++++++++++++++++---------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 2217c0176..a93f9ae08 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -4,9 +4,10 @@ the DLWoodruff fork. Proceeding deliberately ("slowly"): requirements confirmed and the major design questions resolved — decision-logic mechanism (§5), policy file + path selection (§5.1), effort tiers (§5.2), bundle sizing & the EF gate -(§5.3), `--inspect-only` (§5.4), rank allocation (§5.5), and the PR1 validator -(§8). The first dated policy file is committed. No production library code yet — -only an uncommitted interpreter *sketch* (§7). +(§5.3), `--inspect-only` (§5.4), rank allocation (§5.5), and the PR1 tooling — +validator (§8) and effort calibrator (§9). The first dated policy file is +committed. No production library code yet — only an uncommitted interpreter +*sketch* (§7). **Author:** dlw (captured with Claude Code assistance) **Last updated:** 2026-06-28 @@ -180,9 +181,11 @@ program does not parse focus from the name, and no machine-read `focus` field is needed. A user selects a focus simply by passing that file's path. The run **logs which policy file (and `policy_version`) it used**, for reproducibility. -**v1 schema** (see the file for the authoritative, self-documenting copy; every -threshold is flagged `_cold_start_guess` and is a placeholder to be tuned with -data): +**v1 schema** (see the file for the authoritative, self-documenting copy). The +*structure* is authored; the *numbers* are produced by the calibration tool (§9) +on the example set — reproducible, not hand-guesses. (The current design-phase +file still carries hand-guesses flagged `_cold_start_guess`; PR1 replaces them +with calibration output.) Schema keys: | Key | Purpose | |---|---| @@ -408,8 +411,8 @@ to inform. The widest cylinder's rank count governs the bundling problem against an absolute **EF budget** — base `ef_effort_budget`, plus `ef_target_seconds`, minus the count rule `ef_if_num_scens_at_most`. - **Dated data files — generation mechanism is the effort-calibration tool - (§9)** (fits coefficients + an effort→seconds scale from a benchmark corpus); - the corpus, versioning, and shipping cadence are **still open**. + (§9)**, in PR1 (calibrated on the examples); a broader benchmark corpus, + versioning, and shipping cadence are **still open**. - **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. --- @@ -428,8 +431,12 @@ smoke-tested; environment/model probing and apply-to-`Config` are stubbed. (equivalent command line + post-run suggestions), apply-to-`Config`, the `probe_scenarios` knob, the `--inspect-only` dry run (§5.4, incl. the shared `verify_instantiation`), the **policy-file validator (§8)** with its CI-gating - layers wired into CI, quick-start docs, and tests. Ships the default on-ramp - and the no-instantiation escape hatch in one PR. + layers wired into CI, the **effort-calibration tool (§9)** — whose output *is* + the shipped policy's effort numbers (calibrated on the examples, not + hand-guesses) — quick-start docs, and tests. Ships the default on-ramp and the + no-instantiation escape hatch in one **large but cohesive** PR: interpreter + + validator + calibrator together make the shipped policy both *runnable* and + *assessable*. - **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds iteration/time-limit defaults and solve-cost-aware bundling. @@ -530,8 +537,9 @@ report all ship in PR1. ## 9. Effort-calibration tool (future; data-tuning side) -The `effort_scaling` coefficients and the effort budgets ship as abstract -cold-start guesses (§5.3) in arbitrary "effort units." A separate **calibration +The `effort_scaling` coefficients and the effort budgets are abstract by +construction (arbitrary "effort units"); left as hand-guesses they are +**unassessable** — reviewers would be reviewing noise (§5.3). A **calibration tool** turns them into data-tuned, *interpretable* values: - **Fit the coefficients.** Run timed solves across a benchmark spread (varying @@ -552,6 +560,10 @@ calibrated scale is per reference machine/solver and approximate. The **plus** tier effectively *re-calibrates per session* via its measured single-scenario time (`t₁`); **base** relies on this offline calibration. -**Status:** future tool (not PR1). The on-ramp ships with cold-start -coefficients; calibration produces the first data-tuned dated policy once a -benchmark corpus exists. +**Status: PR1.** Reviewers can't assess a policy whose effort coefficients and +budgets are *wild guesses*, so the calibration tool ships in PR1 and **the v1 +policy's numbers are its output** — an initial calibration on the **example set** +(the same timed solves the validator's run-tier performs, so they share +infrastructure), reproducible by re-running it. Preliminary but principled, and +far better than hand-guesses; a broader benchmark corpus (more models / machines) +is the future refinement that later dated files fold in. From c9ca7ef2b2e39b756d090293c9a6a095bb38aabe Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:57:14 -0700 Subject: [PATCH 25/34] Out-of-the-box: note that --out-of-the-box-plus is not a tuning tool plus is still OOTB -- the same one-shot decisions as base, just with a richer fact base (measured solve time, LP/integrality gap). It does not iterate, search a parameter space, or auto-tune to optimality; it's "out-of-the-box with more information," not an autotuner. Noted in sec.5.2 and tied to the sec.1 non-goal; distinguished from the separate offline calibration tool (sec.9). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index a93f9ae08..44e670532 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -62,7 +62,8 @@ explanation of what was chosen and how to do better. - Replacing the explicit cylinder command line for expert users. OOTB is an *on-ramp*, and it deliberately emits the explicit command line it chose. - Tuning convergence parameters to optimality. OOTB aims for *defensible*, - not *optimal*, configurations. + not *optimal*, configurations — this includes `--out-of-the-box-plus`, which is + OOTB *with more information*, not an autotuner (§5.2). - Running a solver to make decisions in the default path. Trial solves are confined to the opt-in `plus` tier (§5.2); minus and base never solve. @@ -239,6 +240,13 @@ to the tier. | base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int` | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; effort-budgeted bundle sizing (§5.3) | | plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | +**`plus` is still OOTB, not a tuning tool.** It makes the *same* one-shot +decisions as base — only with a richer fact base (measured single-scenario solve +time, LP-relaxation / integrality gap). It does **not** iterate, search a +parameter space, or auto-tune to optimality (a non-goal, §1); think *"out-of-the- +box with more information,"* not an autotuner. (Offline coefficient fitting is a +separate concern — the calibration tool, §9.) + **Mechanism.** The three flags set one internal `ootb_effort` level (`minus`/`base`/`plus`); at most one may be supplied. Each takes an **optional policy-file path** — declared `domain=str, default=None, From c3718a145643108118e3be7ff0baa606b645f991 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 15:59:52 -0700 Subject: [PATCH 26/34] Out-of-the-box: commit the interpreter sketch (decision logic complete, wiring stubbed) Baseline for PR1 coding. The pure (facts, policy) -> Decision logic is complete and smoke-tested: recommend() (solver -> EF gate w/ effort budget + forced-decomp override -> small-core/widen spoke roster -> unbalanced flex-rank split -> effort-budgeted bundle sizing -> per-concern option_categories w/ superseded_by), _effort / _pick_spb_by_effort, _allocate_ranks, computed suggestion generators, load_policy (newest dated), command_line, DECOMPOSITION_FLAGS. Stubbed (TODO, the wiring): gather_facts probing (_detect_num_scens, size profile, verify_instantiation, user_flags), apply Decision onto Config, the run, and the plus measure-and-scale hook. ruff clean; imports clean. See doc/designs/out_of_the_box_design.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- mpisppy/generic/out_of_the_box.py | 542 ++++++++++++++++++++++++++++++ 1 file changed, 542 insertions(+) create mode 100644 mpisppy/generic/out_of_the_box.py diff --git a/mpisppy/generic/out_of_the_box.py b/mpisppy/generic/out_of_the_box.py new file mode 100644 index 000000000..82038c395 --- /dev/null +++ b/mpisppy/generic/out_of_the_box.py @@ -0,0 +1,542 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) auto-configuration -- SKETCH, not yet wired. + +This module is the thin Python *interpreter* described in +``doc/designs/out_of_the_box_design.md`` (sec. 5/5.1). It turns: + + * a set of FACTS about the environment (MPI ranks, installed solvers, + optional cores/memory) and the model (scenario count, stage structure), and + * a dated declarative POLICY file (``ootb_policies/ootb_policy_*.json``) + +into a recommended mpi-sppy configuration, a human-readable reason for every +choice, a post-run "Suggestions" list, and the equivalent explicit command +line. + +Design commitments this sketch embodies: + * USER OPTIONS ALWAYS WIN -- ``choose()`` defers to anything in + ``facts.user_flags`` (requirement 0). + * The decision logic is plain, ordered, readable Python -- no rules engine, + no neural net. The *numbers* live in the policy file, not here. + * Every decision records WHY, so transparency (requirement 4) falls out. + +What is COMPLETE here (study this): ``recommend()`` and its helpers, the named +predicates, and command-line assembly -- all pure functions of (facts, policy), +so you can hand-build a ``Facts`` and call ``recommend()`` to see the choices. + +What is STUBBED here (the wiring, deliberately deferred): the environment/model +probing in ``gather_facts()`` and applying the ``Decision`` back onto a +``Config`` object. These touch MPI/Pyomo/Config and are marked ``TODO``. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass, field + +# --------------------------------------------------------------------------- +# Data carriers +# --------------------------------------------------------------------------- + + +@dataclass +class Facts: + """Everything OOTB is allowed to look at (besides the policy).""" + + module_name: str + num_ranks: int + available_solvers: set[str] # names for which SolverFactory(.).available() + num_scens: int + # probe size profile -- populated at the base/plus tiers, None at minus + vars_int: int | None = None # integer/binary vars per scenario + vars_cont: int | None = None # continuous vars per scenario + nonants_total: int | None = None # first-stage (nonanticipative) vars + nonants_int: int | None = None # integer nonants + multistage: bool = False + num_cores: int | None = None # best effort; may be None + memory_gb: float | None = None # best effort; may be None + under_slurm: bool = False + user_flags: set[str] = field(default_factory=set) # CLI flags the user set + + +@dataclass +class ChosenArg: + flag: str # e.g. "--lagrangian" or "--solver-name" + value: str | None # None => boolean flag + reason: str + + +@dataclass +class Decision: + run_ef: bool = False + ef_reason: str | None = None # "min_ranks" | "few_scens" | "user" + chosen_solver: str | None = None + num_cylinders: int = 1 # hub + spokes actually configured + intra_ranks: int = 1 # widest cylinder's rank count + rank_split: dict = field(default_factory=dict) # cylinder -> ranks (flex) + args: list[ChosenArg] = field(default_factory=list) # what OOTB added + notes: list[str] = field(default_factory=list) # full reasoning trace + suggestions: list[str] = field(default_factory=list) # filled AFTER the run + + def command_line(self, num_ranks: int) -> str: + """The explicit command the OOTB choices are equivalent to (req. 4).""" + parts = [ + f"mpiexec -np {num_ranks} python -m mpi4py -m mpisppy.generic_cylinders", + ] + for a in self.args: + parts.append(a.flag if a.value is None else f"{a.flag} {a.value}") + return " ".join(parts) + + +# Flags that mean "the user wants a decomposition": any wired spoke or a +# non-default hub. If the user set one of these AND has >= the rank floor, OOTB +# must NOT substitute the EF, even for a small problem (requirement 0). This is +# the generic_cylinders vocabulary -- a fact, not a focus preference -- so it +# lives in code, and the validator checks it against the actual CLI flags. +DECOMPOSITION_FLAGS = frozenset({ + # wired spokes + "--lagrangian", "--fwph", "--ph-dual", "--ph-xfeas-spoke", "--relaxed-ph", + "--subgradient", "--reduced-costs", "--xhatshuffle", "--xhatxbar", + "--xhatlshaped", + # non-default hubs + "--APH", "--subgradient-hub", "--fwph-hub", "--ph-primal-hub", + "--lshaped-hub", "--cg-hub", "--dualcg-hub", +}) + + +# --------------------------------------------------------------------------- +# The interpreter: pure (facts, policy) -> Decision [COMPLETE] +# --------------------------------------------------------------------------- + + +def recommend(facts: Facts, policy: dict) -> Decision: + """Apply the policy to the facts. Ordered, each step records its reason.""" + d = Decision() + + def choose(flag: str, value: str | None, reason: str) -> bool: + # requirement 0: never override an explicit user choice. + if flag in facts.user_flags: + d.notes.append(f"{flag}: kept user's value (OOTB defers)") + return False + d.args.append(ChosenArg(flag, value, reason)) + shown = flag if value is None else f"{flag} {value}" + d.notes.append(f"{shown}: {reason}") + return True + + # --- step 1: solver ----------------------------------------------------- + sp = policy["solver"] + if "--solver-name" in facts.user_flags: + d.chosen_solver = None # user's solver stands; we don't know its name here + d.notes.append("--solver-name: kept user's value (OOTB defers)") + else: + for name in sp["preference_order"]: + if name in facts.available_solvers: + d.chosen_solver = name + choose("--solver-name", name, + f"first available in preference order ({name})") + break + if d.chosen_solver is None: + d.notes.append("WARNING: no known solver detected; user must supply one") + + # LP/MIP-only solvers cannot take the quadratic PH prox term. + if d.chosen_solver in sp["lp_mip_only_force_linearize_prox"]: + choose("--linearize-proximal-terms", None, + f"{d.chosen_solver} is LP/MIP-only; prox must be linearized") + + # --- step 2: EF gate ---------------------------------------------------- + # Reuses the bundle effort() model on the WHOLE problem (all scenarios as one + # model) vs an ABSOLUTE EF budget. base: effort(num_scens) <= ef_effort_budget; + # plus would use ef_target_seconds via a measured t1 (stubbed); minus (no + # size profile): the count rule. + ef = policy["ef_fallback"] + have_profile = facts.vars_cont is not None or facts.vars_int is not None + if "--EF" in facts.user_flags: + d.run_ef, d.ef_reason = True, "user" + d.notes.append("--EF: user requested") + elif facts.num_ranks < ef["min_ranks_for_decomposition"]: + d.run_ef, d.ef_reason = True, "min_ranks" + choose("--EF", None, + f"only {facts.num_ranks} ranks; decomposition needs " + f">= {ef['min_ranks_for_decomposition']}") + elif facts.user_flags & DECOMPOSITION_FLAGS: + # User explicitly asked for a decomposition and has enough ranks, so we + # never substitute the EF -- even for a small problem (requirement 0). + forced = ", ".join(sorted(facts.user_flags & DECOMPOSITION_FLAGS)) + d.notes.append(f"EF gate skipped: user requested decomposition ({forced}) " + f"with >= {ef['min_ranks_for_decomposition']} ranks") + elif have_profile: + # base/plus: EF when the whole monolith is within the absolute EF budget. + whole = _effort(facts.num_scens, facts, policy["effort_scaling"]) + if whole <= ef["ef_effort_budget"]: + d.run_ef, d.ef_reason = True, "small_effort" + choose("--EF", None, + f"whole-problem effort {whole:.0f} <= EF budget " + f"{ef['ef_effort_budget']}") + elif facts.num_scens <= ef["ef_if_num_scens_at_most"]: + # minus: no profile -> count rule + d.run_ef, d.ef_reason = True, "few_scens" + choose("--EF", None, + f"only {facts.num_scens} scenarios; too few to decompose") + + if d.run_ef: + # EF bypasses the hub/spoke system entirely; skip spokes + bundling. + d.num_cylinders = 1 + return d + + # --- step 3: spoke roster -- small core, widened by ranks -------------- + # Take the minimal core (>=1 outer + >=1 inner). Add further ladder rungs + # only while each cylinder keeps >= min_ranks_per_cylinder ranks (coarse, + # uniform gate) and we stay <= max_cylinders -- prefer giving cylinders + # width over piling on weaker spokes (6 ranks -> 3 cylinders, not 6). + ladder = policy["spoke_ladder"] + max_cyl = ladder["max_cylinders"] + min_rpc = policy["rank_allocation"]["min_ranks_per_cylinder"] + all_rungs = sorted(ladder["rungs"], key=lambda r: r["priority"]) + need = dict(ladder["core_roster_min"]) # e.g. {"outer": 1, "inner": 1} + chosen = [] + for r in all_rungs: # minimal core + if need.get(r["bound"], 0) > 0: + chosen.append(r) + need[r["bound"]] -= 1 + for r in all_rungs: # widen-aware additions + if r in chosen: + continue + cyl_if_added = 2 + len(chosen) # hub + chosen + this rung + if cyl_if_added > max_cyl or facts.num_ranks // cyl_if_added < min_rpc: + break + chosen.append(r) + for r in chosen: + choose(r["flag"], None, f"spoke ({r['bound']}, priority {r['priority']})") + d.num_cylinders = 1 + len(chosen) + + # --- step 4: rank allocation -- UNBALANCED by per-cylinder ratio -------- + # Split ranks across cylinders by ratio (flex-ranks), not uniformly; xhat + # cylinders are cheaper so get a smaller share. CRUDE cold-start: the real + # split depends on subproblem solve cost (a plus-tier refinement). + ra = policy["rank_allocation"] + ratios = {"(hub)": ra["default_rank_ratio"]} + for r in chosen: + ratios[r["flag"]] = ra["rank_ratios"].get(r["flag"], ra["default_rank_ratio"]) + d.rank_split = _allocate_ranks(facts.num_ranks, ratios) + d.intra_ranks = max(d.rank_split.values()) # widest cylinder governs bundling + d.notes.append("rank split (flex-ranks, crude cold-start): " + + ", ".join(f"{k}={v}" for k, v in d.rank_split.items())) + # TODO(wiring): emit the flex-ranks per-cylinder rank flags from rank_split. + + # --- step 5: proper bundling -- how BIG? (design "Bundle sizing") ------- + # minus CANNOT bundle: with no size profile there is no safe way to size a + # bundle, so the minus tier always runs unbundled. + bs = policy["bundle_sizing"] + if not have_profile: # have_profile computed in the EF-gate step above + d.notes.append("bundling: minus tier (no size profile); running unbundled") + elif facts.num_scens >= bs["min_scens_to_consider_bundling"]: + b_min = max(d.intra_ranks, + bs["min_bundles_per_intra_rank"] * d.intra_ranks) + spb = _pick_spb_by_effort( + facts.num_scens, b_min, facts, policy["effort_scaling"], + bs["base_max_hardness_vs_single_scenario"]) + if spb is not None: + nb = facts.num_scens // spb + choose("--scenarios-per-bundle", str(spb), + f"{facts.num_scens} scenarios -> {nb} bundles of {spb} " + f"(effort <= {bs['base_max_hardness_vs_single_scenario']}x a " + f"single scenario, >= {b_min} bundles)") + else: + d.notes.append( + f"bundling: no divisor of {facts.num_scens} qualifies " + f"(>= {b_min} bundles within budget); running unbundled" + ) + + # --- step 6: extra options by concern; any superseded_by flag defers ---- + # Per-concern override: OOTB backs off a whole concern (e.g. its rho setter) + # if the user set ANY equivalent flag, not just the identical one -- mp-sppy + # allows only one rho setter, so stacking would be a hard error. + for name, cat in policy.get("option_categories", {}).items(): + if name.startswith("_"): + continue + if any(f in facts.user_flags for f in cat.get("superseded_by", [cat["flag"]])): + d.notes.append(f"{name}: superseded by a user option; OOTB defers") + else: + choose(cat["flag"], cat.get("value"), f"policy option ({name})") + # catch-all: per-flag override (superseded_by defaults to the flag itself) + for opt in policy.get("additional_options", {}).get("options", []): + if any(f in facts.user_flags for f in opt.get("superseded_by", [opt["flag"]])): + d.notes.append(f"{opt['flag']}: superseded by a user option; OOTB defers") + else: + choose(opt["flag"], opt.get("value"), "policy additional option") + + return d + + +def _effort(spb: int, facts: Facts, scaling: dict) -> float: + """Modeled solve effort of a bundle of `spb` scenarios, from the probe size + profile (facts.vars_cont/vars_int/nonants_int) and the policy effort_scaling + shape. Continuous ~linear, integers superlinear (int_exponent > 1), integer + nonants a fixed per-bundle coupling cost. See design "Bundle sizing".""" + cont = (facts.vars_cont or 0) * spb + nint = (facts.vars_int or 0) * spb + return (scaling["cont_coeff"] * cont + + scaling["int_weight"] * (nint ** scaling["int_exponent"]) + + scaling["int_nonant_coeff"] * (facts.nonants_int or 0)) + + +def _pick_spb_by_effort(num_scens: int, min_bundles: int, facts: Facts, + scaling: dict, max_hardness: float) -> int | None: + """base/plus sizer: the LARGEST scenarios_per_bundle that divides num_scens, + leaves >= min_bundles bundles, and keeps a bundle's modeled effort within + `max_hardness` x a single scenario (the relative, unit-free budget). The plus + tier feeds the same function a time-calibrated effective hardness. Returns + None when nothing past the degenerate spb==1 qualifies.""" + e1 = _effort(1, facts, scaling) + best = None + for spb in range(2, num_scens + 1): # spb==1 is "no bundling" + if num_scens % spb or num_scens // spb < min_bundles: + continue + if e1 > 0 and _effort(spb, facts, scaling) / e1 > max_hardness: + continue + if best is None or spb > best: # largest = max amortization + best = spb + return best + + +def _allocate_ranks(total: int, ratios: dict) -> dict: + """Split `total` ranks across cylinders by ratio (flex-ranks), flooring at 1 + rank each and distributing the remainder by largest fractional remainder. + CRUDE cold-start; the real split depends on subproblem solve cost (design + 'Rank allocation').""" + names = list(ratios) + if total <= len(names): + return {nm: 1 for nm in names} # too few to weight; 1 each + extra = total - len(names) # ranks above the 1-each floor + s = sum(ratios.values()) or 1.0 + raw = {nm: extra * ratios[nm] / s for nm in names} + alloc = {nm: 1 + int(raw[nm]) for nm in names} + leftover = total - sum(alloc.values()) + for nm in sorted(names, key=lambda k: raw[k] - int(raw[k]), reverse=True)[:leftover]: + alloc[nm] += 1 + return alloc + + +# --------------------------------------------------------------------------- +# Suggestions -- MOSTLY COMPUTED [COMPLETE] +# Each generator inspects facts/decision/run-outcome and COMPUTES a message +# (with live values) or returns None. The prose lives here in code; the policy +# only toggles (suggestions.disabled) or tunes them. This is the diagnostics +# layer -- deliberately distinct from the data-driven DECISIONS (design 5). +# Generators run in list order (priority); add outcome-based ones freely -- +# they receive the post-run `outcome`. +# --------------------------------------------------------------------------- + + +def _sg_ran_ef_few_ranks(d, facts, policy, outcome): + if d.run_ef and d.ef_reason == "min_ranks": + need = policy["ef_fallback"]["min_ranks_for_decomposition"] + return (f"Ran the monolithic EF because only {facts.num_ranks} MPI " + f"rank(s) were available; with >= {need} ranks OOTB would " + f"decompose (hub + bound spokes).") + return None + + +def _sg_no_persistent_solver(d, facts, policy, outcome): + s = d.chosen_solver + if s is not None and not s.endswith("_persistent"): + return (f"Chosen solver '{s}' has no persistent interface available; " + f"'{s}_persistent' would warm-start subproblems and is usually " + f"much faster for PH.") + return None + + +def _sg_linearized_prox(d, facts, policy, outcome): + if d.chosen_solver in policy["solver"]["lp_mip_only_force_linearize_prox"]: + return (f"'{d.chosen_solver}' is LP/MIP-only, so the PH prox is being " + f"linearized; a QP-capable solver (gurobi/cplex/xpress, or ipopt " + f"for continuous models) avoids the approximation.") + return None + + +def _sg_more_ranks(d, facts, policy, outcome): + cap = policy["spoke_ladder"]["max_cylinders"] + if not d.run_ef and d.num_cylinders < cap: + return (f"Only {d.num_cylinders} cylinders configured (cap {cap}); more " + f"MPI ranks would add bound-tightening spokes or intra-cylinder " + f"parallelism.") + return None + + +def _sg_from_outcome(d, facts, policy, outcome): + # Example of an outcome-based (post-run) computed suggestion; inert until the + # run captures an outcome (None in the current sketch). + if outcome and outcome.get("converged") is False: + return (f"PH stopped at {outcome.get('iterations')} iterations with a " + f"{outcome.get('rel_gap', 0):.0%} gap; consider raising " + f"--max-iterations or adding a tighter bound spoke.") + return None + + +# generators in priority order (lower first) +SUGGESTION_GENERATORS = [ + _sg_ran_ef_few_ranks, + _sg_no_persistent_solver, + _sg_linearized_prox, + _sg_more_ranks, + _sg_from_outcome, +] + + +def make_suggestions(d: Decision, facts: Facts, policy: dict, + outcome: dict | None = None) -> list[str]: + """Build the post-run "Suggestions" list (req. 4) by running the computed + generators in priority order, skipping any named in suggestions.disabled. + Called AFTER the run so generators may use `outcome` (convergence, gap, + iters, time).""" + disabled = set(policy.get("suggestions", {}).get("disabled", [])) + out = [] + for gen in SUGGESTION_GENERATORS: + if gen.__name__ in disabled: + continue + msg = gen(d, facts, policy, outcome) + if msg: + out.append(msg) + return out + + +# --------------------------------------------------------------------------- +# Policy loading [COMPLETE] +# --------------------------------------------------------------------------- + + +def _policies_dir() -> str: + return os.path.join(os.path.dirname(__file__), "ootb_policies") + + +def load_policy(policy_file: str | None = None) -> dict: + """Load a policy. Default: newest dated file in ootb_policies/. The dated + filename sorts lexically by ISO date, so max() is the newest.""" + if policy_file is None: + d = _policies_dir() + candidates = sorted(f for f in os.listdir(d) + if f.startswith("ootb_policy_") and f.endswith(".json")) + if not candidates: + raise FileNotFoundError(f"no OOTB policy files in {d}") + policy_file = os.path.join(d, candidates[-1]) + with open(policy_file) as fp: + return json.load(fp) + + +# --------------------------------------------------------------------------- +# Environment/model probing + apply-to-Config [STUBBED -- the wiring] +# --------------------------------------------------------------------------- + + +def _detect_num_ranks() -> int: + from mpisppy import MPI + return MPI.COMM_WORLD.Get_size() + + +def _inspect_ranks(cfg) -> int: + """Ranks to PLAN for. With --inspect-only N, use N so HPC users can get a + recommended command line for a target job size WITHOUT launching that many + ranks; otherwise the actual detected size. (N rides on --inspect-only, the + only flag that carries it; everything else -- solvers, model size -- still + comes from the real, possibly small, session.)""" + io = cfg.get("inspect_only", None) + if io not in (None, "", "detected"): + return int(io) + return _detect_num_ranks() + + +def _detect_available_solvers(candidates: list[str]) -> set[str]: + import pyomo.environ as pyo + found = set() + for name in candidates: + try: + if pyo.SolverFactory(name).available(exception_flag=False): + found.add(name) + except Exception: + pass + return found + + +def _detect_num_scens(module, cfg) -> int: + # TODO(wiring): mirror mpisppy/generic/parsing.py::name_lists -- + # cfg.num_scens, else prod(cfg.branching_factors), + # else len(module.scenario_names_creator(None)). + raise NotImplementedError("num_scens probing not wired yet") + + +def verify_instantiation(module, cfg): + """Build a single scenario to confirm the model instantiates -- a cheap + model smoke-test. SHARED CODE: used by --inspect-only when given WITHOUT + --out-of-the-box, and by the base/plus probe in gather_facts (which also + reads the size profile off the built model). STUB: wiring deferred.""" + # TODO(wiring): name = module.scenario_names_creator(1, cfg=cfg)[0] + # model = module.scenario_creator(name, **kwargs) # raises if it can't build + # return the size profile (vars_int, vars_cont, nonants_total, nonants_int) + raise NotImplementedError("scenario instantiation/verification not wired yet") + + +def gather_facts(module, cfg) -> Facts: + """Assemble Facts from the environment, the model module, and cfg. + STUB: the pieces below marked TODO are the integration work.""" + policy = load_policy() + solvers = _detect_available_solvers(policy["solver"]["preference_order"]) + facts = Facts( + module_name=cfg.get("module_name", ""), + num_ranks=_inspect_ranks(cfg), + available_solvers=solvers, + num_scens=_detect_num_scens(module, cfg), + num_cores=os.cpu_count(), + under_slurm=("SLURM_JOB_ID" in os.environ), + # TODO(wiring): memory_gb (OS/SLURM-dependent), multistage flag, + # user_flags (which options the user explicitly set on cfg), and -- at + # the base/plus tiers -- the probe size profile (vars_int, vars_cont, + # nonants_total, nonants_int) by instantiating probe_scenarios scenarios. + user_flags=set(), + ) + return facts + + +def out_of_the_box(module, cfg): + """Top-level entry (STUB). Probe, recommend, report the config up front, + run, then print Suggestions afterward.""" + facts = gather_facts(module, cfg) + policy = load_policy() + decision = recommend(facts, policy) + + # report the configuration up front so the user sees what is running + # (rank-0 printing is the caller's concern). + print(f"[out-of-the-box] using policy {policy['policy_version']}") + for note in decision.notes: + print(f" - {note}") + print(f"[out-of-the-box] equivalent command line:\n " + f"{decision.command_line(facts.num_ranks)}") + + if cfg.get("inspect_only", None) is not None: + # --inspect-only: inspection is already done -- instantiation, and any + # plus calibration solves, happened in gather_facts (resolution B) -- + # so just skip the production run. facts.num_ranks may be an assumed + # count (--inspect-only N) for HPC planning. + print(f"[out-of-the-box] --inspect-only: planning for {facts.num_ranks} " + f"ranks; skipping the production run") + outcome = None + else: + # TODO(wiring): apply decision.args onto cfg, then run via the normal + # generic_cylinders driver path (EF vs cylinders); capture an `outcome` + # (convergence, gap, iters, time) to enrich the suggestions. + outcome = None + + # Suggestions are written AFTER the run (req. 4), labelled "Suggestions". + decision.suggestions = make_suggestions(decision, facts, policy, outcome) + if decision.suggestions: + print("[out-of-the-box] Suggestions:") + for s in decision.suggestions: + print(f" * {s}") + + return decision From 21ced1bf923d641ae06d3bc9800a451c675f77a6 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 16:00:28 -0700 Subject: [PATCH 27/34] Out-of-the-box: status header reflects the now-committed sketch The interpreter sketch is committed (decision logic complete, wiring stubbed), so the status line no longer calls it uncommitted; next step is turning it into PR1 code. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index 44e670532..b491e8ef8 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -6,8 +6,8 @@ and the major design questions resolved — decision-logic mechanism (§5), poli file + path selection (§5.1), effort tiers (§5.2), bundle sizing & the EF gate (§5.3), `--inspect-only` (§5.4), rank allocation (§5.5), and the PR1 tooling — validator (§8) and effort calibrator (§9). The first dated policy file is -committed. No production library code yet — only an uncommitted interpreter -*sketch* (§7). +committed, as is an interpreter *sketch* (§7) whose decision logic is complete +with the environment wiring stubbed. Next: turn the sketch into PR1 code. **Author:** dlw (captured with Claude Code assistance) **Last updated:** 2026-06-28 From ba3f0a52f497e58ea0fccf4ee95b96d55edb095f Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 16:55:55 -0700 Subject: [PATCH 28/34] Out-of-the-box: wire the interpreter -- minus + base tiers run end-to-end Turn the OOTB sketch into working code. The pure recommend(facts, policy) logic was already complete; this commit adds the environment/model wiring and hooks it into the generic_cylinders driver, so --out-of-the-box-minus and --out-of-the-box (base) now actually configure and run a model. - config.ootb_args(): the three mutually-exclusive tier flags plus --inspect-only, each an optional-value flag (str + nargs='?', const='') so a bare flag uses the shipped default policy and a value names a policy file. Wired into generic/parsing.py. - out_of_the_box.py wiring: effort/policy resolution, gather_facts (structural facts always; a one-scenario probe size profile at base/plus), verify_instantiation (the shared probe / model smoke-test), apply_decision (Decision -> cfg so the normal driver path runs it), and the configure/report_suggestions entry points (rank-0 printing). - generic_cylinders.py: run configure() right after arg parsing (mutating cfg), short-circuit on --inspect-only, and print the post-run Suggestions. Solver-key split (raised as a design issue): the EF path reads cfg.EF_solver_name and the decomposition path cfg.solver_name. recommend() now settles the solver NAME first and emits the path-appropriate flag once the EF gate decides; a naive user's --solver-name is carried over to --EF-solver-name when OOTB runs the EF (belt-and-suspenders guarantee in apply_decision too). Rank allocation now emits the per-spoke ---rank-ratio flags and mirrors WheelSpinner.apportion_ranks for the bundling #bundles>=intra_ranks floor. Policy: add a base_rho option category (--default-rho 1). PH needs a base rho; --grad-rho refines but does not supply one and (unlike sep/coeff/sensi) is not auto-defaulted, so a --grad-rho decomposition without --default-rho was a hard error. Verified on farmer: EF (few ranks / small effort / few scens), decomposition (minus + base), proper bundling (base, forced-decomp), --inspect-only, and the solver-name->EF mapping all run. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../ootb_policies/ootb_policy_2026-06-28.json | 7 +- mpisppy/generic/out_of_the_box.py | 441 +++++++++++++----- mpisppy/generic/parsing.py | 1 + mpisppy/generic_cylinders.py | 20 + mpisppy/utils/config.py | 58 +++ 5 files changed, 406 insertions(+), 121 deletions(-) diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 692f59bc3..26c99c089 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -99,6 +99,11 @@ "option_categories": { "_comment": "Named option blocks OOTB applies by default, one per CONCERN, beyond the structural choices (solver / EF / spokes / bundling). Each is applied UNLESS the user already set any flag in its 'superseded_by' list -- the robust, per-concern form of 'user options win' (the user may address a concern with a DIFFERENT flag than OOTB's default). 'value' null = a boolean flag. These are decomposition-run options (skipped on the EF path). Foci may change them. All cold-start.", + "base_rho": { + "flag": "--default-rho", "value": "1", + "superseded_by": ["--default-rho"], + "_note": "PH needs a base rho value. The grad-rho setter REFINES rho but does not itself supply a base, and (unlike sep/coeff/sensi rho) mpi-sppy does not auto-default it, so a decomposition with --grad-rho and no --default-rho is a hard error. Set a sane base here; the rho setter adapts from it." + }, "rho_setter": { "flag": "--grad-rho", "value": null, "superseded_by": ["--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho", "--reduced-costs-rho"], @@ -117,7 +122,7 @@ "superseded_by": ["--dynamic-rho-primal-crit", "--dynamic-rho-dual-crit"], "_note": "Boolean flag (no argument needed); its paired threshold dynamic_rho_primal_thresh defaults to 0.1. Requires an active rho setter (config errors otherwise), which OOTB's default --grad-rho provides." }, - "_cold_start_guess": ["rho_setter", "termination", "max_iterations", "dynamic_rho"] + "_cold_start_guess": ["base_rho", "rho_setter", "termination", "max_iterations", "dynamic_rho"] }, "additional_options": { diff --git a/mpisppy/generic/out_of_the_box.py b/mpisppy/generic/out_of_the_box.py index 82038c395..26e426cdc 100644 --- a/mpisppy/generic/out_of_the_box.py +++ b/mpisppy/generic/out_of_the_box.py @@ -6,41 +6,58 @@ # All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for # full copyright and license information. ############################################################################### -"""Out-of-the-box (OOTB) auto-configuration -- SKETCH, not yet wired. +"""Out-of-the-box (OOTB) auto-configuration. This module is the thin Python *interpreter* described in ``doc/designs/out_of_the_box_design.md`` (sec. 5/5.1). It turns: * a set of FACTS about the environment (MPI ranks, installed solvers, - optional cores/memory) and the model (scenario count, stage structure), and + optional cores/memory) and the model (scenario count, stage structure, and + -- at the base/plus tiers -- a probe size profile), and * a dated declarative POLICY file (``ootb_policies/ootb_policy_*.json``) into a recommended mpi-sppy configuration, a human-readable reason for every choice, a post-run "Suggestions" list, and the equivalent explicit command -line. +line. The recommendation is applied back onto the ``Config`` object so the +normal ``generic_cylinders`` driver path executes it. -Design commitments this sketch embodies: - * USER OPTIONS ALWAYS WIN -- ``choose()`` defers to anything in +Design commitments: + * USER OPTIONS ALWAYS WIN -- ``recommend()`` defers to anything in ``facts.user_flags`` (requirement 0). * The decision logic is plain, ordered, readable Python -- no rules engine, no neural net. The *numbers* live in the policy file, not here. * Every decision records WHY, so transparency (requirement 4) falls out. -What is COMPLETE here (study this): ``recommend()`` and its helpers, the named -predicates, and command-line assembly -- all pure functions of (facts, policy), -so you can hand-build a ``Facts`` and call ``recommend()`` to see the choices. - -What is STUBBED here (the wiring, deliberately deferred): the environment/model -probing in ``gather_facts()`` and applying the ``Decision`` back onto a -``Config`` object. These touch MPI/Pyomo/Config and are marked ``TODO``. +Pure vs. wired: + * Pure functions of (facts, policy): ``recommend()`` and its helpers, the + suggestion generators, ``load_policy``. Hand-build a ``Facts`` and call + ``recommend()`` to see the choices (this is what the validator's + synthetic-facts layer does). + * Wiring (touches MPI/Pyomo/Config): ``gather_facts``, ``verify_instantiation``, + ``apply_decision``, and the ``configure``/``report_suggestions`` entry + points the driver calls. """ from __future__ import annotations import json +import math import os +import sys from dataclasses import dataclass, field +# --------------------------------------------------------------------------- +# Effort tiers: the three OOTB flags map to one internal level. The flag name +# is the cfg key (dashes -> underscores); the value is an optional policy path. +# --------------------------------------------------------------------------- + +EFFORT_FLAGS = { + "out_of_the_box_minus": "minus", + "out_of_the_box": "base", + "out_of_the_box_plus": "plus", +} + + # --------------------------------------------------------------------------- # Data carriers # --------------------------------------------------------------------------- @@ -54,12 +71,15 @@ class Facts: num_ranks: int available_solvers: set[str] # names for which SolverFactory(.).available() num_scens: int + effort: str = "base" # "minus" | "base" | "plus" # probe size profile -- populated at the base/plus tiers, None at minus vars_int: int | None = None # integer/binary vars per scenario vars_cont: int | None = None # continuous vars per scenario nonants_total: int | None = None # first-stage (nonanticipative) vars nonants_int: int | None = None # integer nonants multistage: bool = False + branching_factors: list | None = None # for the equivalent command line + user_solver_name: str | None = None # name the user gave (if any) num_cores: int | None = None # best effort; may be None memory_gb: float | None = None # best effort; may be None under_slurm: bool = False @@ -76,7 +96,7 @@ class ChosenArg: @dataclass class Decision: run_ef: bool = False - ef_reason: str | None = None # "min_ranks" | "few_scens" | "user" + ef_reason: str | None = None # "min_ranks" | "few_scens" | "user" ... chosen_solver: str | None = None num_cylinders: int = 1 # hub + spokes actually configured intra_ranks: int = 1 # widest cylinder's rank count @@ -85,11 +105,24 @@ class Decision: notes: list[str] = field(default_factory=list) # full reasoning trace suggestions: list[str] = field(default_factory=list) # filled AFTER the run - def command_line(self, num_ranks: int) -> str: - """The explicit command the OOTB choices are equivalent to (req. 4).""" + def command_line(self, facts: "Facts") -> str: + """The explicit command the OOTB choices are equivalent to (req. 4). + + Anchored with the module and scenario specification so it is runnable + without --out-of-the-box; the OOTB-added flags follow. (Options the user + set explicitly are not repeated here -- they were already on the user's + command line and OOTB left them untouched.) + """ parts = [ - f"mpiexec -np {num_ranks} python -m mpi4py -m mpisppy.generic_cylinders", + f"mpiexec -np {facts.num_ranks} python -m mpi4py -m " + f"mpisppy.generic_cylinders", + f"--module-name {facts.module_name}", ] + if facts.multistage and facts.branching_factors: + parts.append("--branching-factors " + + " ".join(str(b) for b in facts.branching_factors)) + else: + parts.append(f"--num-scens {facts.num_scens}") for a in self.args: parts.append(a.flag if a.value is None else f"{a.flag} {a.value}") return " ".join(parts) @@ -130,25 +163,25 @@ def choose(flag: str, value: str | None, reason: str) -> bool: d.notes.append(f"{shown}: {reason}") return True - # --- step 1: solver ----------------------------------------------------- + # --- step 1: pick the solver NAME --------------------------------------- + # The flag is emitted only once the EF gate decides: --EF-solver-name for the + # EF, --solver-name to decompose (mpi-sppy uses different keys). Here we just + # settle on the name. sp = policy["solver"] - if "--solver-name" in facts.user_flags: - d.chosen_solver = None # user's solver stands; we don't know its name here - d.notes.append("--solver-name: kept user's value (OOTB defers)") + if facts.user_flags & {"--solver-name", "--EF-solver-name"} and facts.user_solver_name: + d.chosen_solver = facts.user_solver_name + d.notes.append(f"solver: kept user's value ({facts.user_solver_name}); " + f"OOTB defers") else: for name in sp["preference_order"]: if name in facts.available_solvers: d.chosen_solver = name - choose("--solver-name", name, - f"first available in preference order ({name})") break if d.chosen_solver is None: d.notes.append("WARNING: no known solver detected; user must supply one") - - # LP/MIP-only solvers cannot take the quadratic PH prox term. - if d.chosen_solver in sp["lp_mip_only_force_linearize_prox"]: - choose("--linearize-proximal-terms", None, - f"{d.chosen_solver} is LP/MIP-only; prox must be linearized") + else: + d.notes.append(f"solver: {d.chosen_solver} " + f"(first available in preference order)") # --- step 2: EF gate ---------------------------------------------------- # Reuses the bundle effort() model on the WHOLE problem (all scenarios as one @@ -187,9 +220,22 @@ def choose(flag: str, value: str | None, reason: str) -> bool: if d.run_ef: # EF bypasses the hub/spoke system entirely; skip spokes + bundling. + # The EF path reads cfg.EF_solver_name (its own key). + if d.chosen_solver is not None: + choose("--EF-solver-name", d.chosen_solver, + f"EF solver ({d.chosen_solver})") d.num_cylinders = 1 return d + # decomposition solver + (if LP/MIP-only) prox linearization. The hub/spoke + # path reads cfg.solver_name. + if d.chosen_solver is not None: + choose("--solver-name", d.chosen_solver, + f"decomposition solver ({d.chosen_solver})") + if d.chosen_solver in sp["lp_mip_only_force_linearize_prox"]: + choose("--linearize-proximal-terms", None, + f"{d.chosen_solver} is LP/MIP-only; the PH prox must be linearized") + # --- step 3: spoke roster -- small core, widened by ranks -------------- # Take the minimal core (>=1 outer + >=1 inner). Add further ladder rungs # only while each cylinder keeps >= min_ranks_per_cylinder ranks (coarse, @@ -218,17 +264,25 @@ def choose(flag: str, value: str | None, reason: str) -> bool: # --- step 4: rank allocation -- UNBALANCED by per-cylinder ratio -------- # Split ranks across cylinders by ratio (flex-ranks), not uniformly; xhat - # cylinders are cheaper so get a smaller share. CRUDE cold-start: the real - # split depends on subproblem solve cost (a plus-tier refinement). + # cylinders are cheaper so get a smaller share. We emit the per-spoke + # ---rank-ratio flags (only when a spoke's ratio differs from the + # default) and let WheelSpinner.apportion_ranks do the real split. CRUDE + # cold-start: the real split depends on subproblem solve cost (a plus-tier + # refinement). ra = policy["rank_allocation"] - ratios = {"(hub)": ra["default_rank_ratio"]} + default_ratio = ra["default_rank_ratio"] + # ratios in cylinder order: hub first (always the default), then spokes. + ratios = [default_ratio] for r in chosen: - ratios[r["flag"]] = ra["rank_ratios"].get(r["flag"], ra["default_rank_ratio"]) - d.rank_split = _allocate_ranks(facts.num_ranks, ratios) - d.intra_ranks = max(d.rank_split.values()) # widest cylinder governs bundling + spoke_ratio = ra["rank_ratios"].get(r["flag"], default_ratio) + ratios.append(spoke_ratio) + if spoke_ratio != default_ratio: + choose(f"{r['flag']}-rank-ratio", _fmt_ratio(spoke_ratio), + f"flex-ranks: cheaper cylinder gets a {spoke_ratio} share " + f"(crude cold-start)") + d.intra_ranks, d.rank_split = _rank_layout(facts.num_ranks, ratios, chosen) d.notes.append("rank split (flex-ranks, crude cold-start): " + ", ".join(f"{k}={v}" for k, v in d.rank_split.items())) - # TODO(wiring): emit the flex-ranks per-cylinder rank flags from rank_split. # --- step 5: proper bundling -- how BIG? (design "Bundle sizing") ------- # minus CANNOT bundle: with no size profile there is no safe way to size a @@ -256,7 +310,7 @@ def choose(flag: str, value: str | None, reason: str) -> bool: # --- step 6: extra options by concern; any superseded_by flag defers ---- # Per-concern override: OOTB backs off a whole concern (e.g. its rho setter) - # if the user set ANY equivalent flag, not just the identical one -- mp-sppy + # if the user set ANY equivalent flag, not just the identical one -- mpi-sppy # allows only one rho setter, so stacking would be a hard error. for name, cat in policy.get("option_categories", {}).items(): if name.startswith("_"): @@ -275,6 +329,11 @@ def choose(flag: str, value: str | None, reason: str) -> bool: return d +def _fmt_ratio(x: float) -> str: + """Render a rank ratio without a trailing ``.0`` (so 0.2 stays 0.2).""" + return str(int(x)) if float(x).is_integer() else str(x) + + def _effort(spb: int, facts: Facts, scaling: dict) -> float: """Modeled solve effort of a bundle of `spb` scenarios, from the probe size profile (facts.vars_cont/vars_int/nonants_int) and the policy effort_scaling @@ -306,22 +365,24 @@ def _pick_spb_by_effort(num_scens: int, min_bundles: int, facts: Facts, return best -def _allocate_ranks(total: int, ratios: dict) -> dict: - """Split `total` ranks across cylinders by ratio (flex-ranks), flooring at 1 - rank each and distributing the remainder by largest fractional remainder. - CRUDE cold-start; the real split depends on subproblem solve cost (design - 'Rank allocation').""" - names = list(ratios) - if total <= len(names): - return {nm: 1 for nm in names} # too few to weight; 1 each - extra = total - len(names) # ranks above the 1-each floor - s = sum(ratios.values()) or 1.0 - raw = {nm: extra * ratios[nm] / s for nm in names} - alloc = {nm: 1 + int(raw[nm]) for nm in names} - leftover = total - sum(alloc.values()) - for nm in sorted(names, key=lambda k: raw[k] - int(raw[k]), reverse=True)[:leftover]: - alloc[nm] += 1 - return alloc +def _rank_layout(total: int, ratios: list, chosen: list) -> tuple: + """Return (intra_ranks, rank_split) for the chosen cylinders. + + `ratios` is hub-first, matching `[hub] + chosen`. When every ratio equals + the first (uniform), WheelSpinner uses the equal-rank split + (total // n_cyl per cylinder); otherwise it apportions by ratio + (largest-remainder, floor of one) -- we mirror that so the bundling + `#bundles >= intra_ranks` floor matches what actually runs. intra_ranks is + the widest cylinder's rank count (it governs the bundle floor).""" + names = ["(hub)"] + [r["flag"] for r in chosen] + if all(x == ratios[0] for x in ratios): + per = max(1, total // len(ratios)) + split = {nm: per for nm in names} + else: + from mpisppy.utils.rank_apportionment import apportion_ranks + counts = apportion_ranks(ratios, total) + split = dict(zip(names, counts)) + return max(split.values()), split # --------------------------------------------------------------------------- @@ -346,7 +407,7 @@ def _sg_ran_ef_few_ranks(d, facts, policy, outcome): def _sg_no_persistent_solver(d, facts, policy, outcome): s = d.chosen_solver - if s is not None and not s.endswith("_persistent"): + if not d.run_ef and s is not None and not s.endswith("_persistent"): return (f"Chosen solver '{s}' has no persistent interface available; " f"'{s}_persistent' would warm-start subproblems and is usually " f"much faster for PH.") @@ -354,7 +415,8 @@ def _sg_no_persistent_solver(d, facts, policy, outcome): def _sg_linearized_prox(d, facts, policy, outcome): - if d.chosen_solver in policy["solver"]["lp_mip_only_force_linearize_prox"]: + if not d.run_ef and \ + d.chosen_solver in policy["solver"]["lp_mip_only_force_linearize_prox"]: return (f"'{d.chosen_solver}' is LP/MIP-only, so the PH prox is being " f"linearized; a QP-capable solver (gurobi/cplex/xpress, or ipopt " f"for continuous models) avoids the approximation.") @@ -370,9 +432,17 @@ def _sg_more_ranks(d, facts, policy, outcome): return None +def _sg_minus_no_bundling(d, facts, policy, outcome): + if facts.effort == "minus" and not d.run_ef: + return ("Ran the --out-of-the-box-minus tier, which instantiates nothing " + "and therefore cannot size proper bundles; --out-of-the-box (base) " + "probes one scenario and can bundle when there are many scenarios.") + return None + + def _sg_from_outcome(d, facts, policy, outcome): - # Example of an outcome-based (post-run) computed suggestion; inert until the - # run captures an outcome (None in the current sketch). + # Outcome-based (post-run) computed suggestion; inert until the run captures + # an outcome (None in the minus/base tiers for now). if outcome and outcome.get("converged") is False: return (f"PH stopped at {outcome.get('iterations')} iterations with a " f"{outcome.get('rel_gap', 0):.0%} gap; consider raising " @@ -386,6 +456,7 @@ def _sg_from_outcome(d, facts, policy, outcome): _sg_no_persistent_solver, _sg_linearized_prox, _sg_more_ranks, + _sg_minus_no_bundling, _sg_from_outcome, ] @@ -417,24 +488,59 @@ def _policies_dir() -> str: def load_policy(policy_file: str | None = None) -> dict: - """Load a policy. Default: newest dated file in ootb_policies/. The dated - filename sorts lexically by ISO date, so max() is the newest.""" - if policy_file is None: + """Load a policy. Default (policy_file None or empty): the newest dated file + with no focus token in ootb_policies/ (``ootb_policy_.json``). The + dated filename sorts lexically by ISO date, so the last is the newest.""" + if not policy_file: d = _policies_dir() - candidates = sorted(f for f in os.listdir(d) - if f.startswith("ootb_policy_") and f.endswith(".json")) + # focus is conveyed by extra filename tokens (ootb_policy_quick_); + # the bare default is ootb_policy_.json -- two underscores exactly. + candidates = sorted( + f for f in os.listdir(d) + if f.startswith("ootb_policy_") and f.endswith(".json") + and f.count("_") == 2 + ) if not candidates: - raise FileNotFoundError(f"no OOTB policy files in {d}") + raise FileNotFoundError(f"no default OOTB policy files in {d}") policy_file = os.path.join(d, candidates[-1]) with open(policy_file) as fp: return json.load(fp) # --------------------------------------------------------------------------- -# Environment/model probing + apply-to-Config [STUBBED -- the wiring] +# Environment/model probing + apply-to-Config [the wiring] # --------------------------------------------------------------------------- +def effort_and_policy(cfg) -> tuple: + """Return (effort, policy_path) from the cfg, or (None, None) if OOTB is off. + + The three tier flags are mutually exclusive; supplying more than one is an + error. ``policy_path`` is the optional value attached to the chosen flag + (empty string -> use the shipped default policy).""" + selected = [(cfg_key, tier) for cfg_key, tier in EFFORT_FLAGS.items() + if cfg.get(cfg_key) is not None] + if not selected: + return None, None + if len(selected) > 1: + names = ", ".join("--" + k.replace("_", "-") for k, _ in selected) + raise RuntimeError( + f"At most one out-of-the-box tier may be given; got {names}.") + cfg_key, tier = selected[0] + policy_path = cfg.get(cfg_key) or None # "" (bare flag) -> default + return tier, policy_path + + +def requested(cfg) -> bool: + """True if any out-of-the-box tier flag was supplied.""" + return any(cfg.get(k) is not None for k in EFFORT_FLAGS) + + +def _rank0() -> bool: + from mpisppy import MPI + return MPI.COMM_WORLD.Get_rank() == 0 + + def _detect_num_ranks() -> int: from mpisppy import MPI return MPI.COMM_WORLD.Get_size() @@ -452,7 +558,7 @@ def _inspect_ranks(cfg) -> int: return _detect_num_ranks() -def _detect_available_solvers(candidates: list[str]) -> set[str]: +def _detect_available_solvers(candidates) -> set: import pyomo.environ as pyo found = set() for name in candidates: @@ -465,78 +571,173 @@ def _detect_available_solvers(candidates: list[str]) -> set[str]: def _detect_num_scens(module, cfg) -> int: - # TODO(wiring): mirror mpisppy/generic/parsing.py::name_lists -- - # cfg.num_scens, else prod(cfg.branching_factors), - # else len(module.scenario_names_creator(None)). - raise NotImplementedError("num_scens probing not wired yet") + """Mirror mpisppy/generic/parsing.py::name_lists: cfg.num_scens, else the + product of the branching factors, else the module's full scenario list.""" + if cfg.get("num_scens") is not None: + return int(cfg.num_scens) + bf = cfg.get("branching_factors") + if bf is not None: + return int(math.prod(bf)) + return len(module.scenario_names_creator(None)) + + +def _user_flags() -> set: + """The set of long CLI flags the user actually typed (--flag, stripped of + any =value). This is how requirement 0 is honored: recommend() defers to any + flag in here. Parsing argv (rather than the post-parse cfg) is what lets us + tell a user-set value apart from a default -- argparse fills defaults for + everything, so the cfg alone cannot say what the user chose.""" + flags = set() + for tok in sys.argv[1:]: + if tok.startswith("--"): + flags.add(tok.split("=", 1)[0]) + return flags + + +def _size_profile(model) -> dict: + """Read a built scenario's size profile: integer vs continuous variable + counts, and nonant (first-stage) counts. Used by the base/plus probe and by + --inspect-only's instantiation check.""" + import pyomo.environ as pyo + vars_int = vars_cont = 0 + for v in model.component_data_objects(pyo.Var, active=True, descend_into=True): + if v.is_continuous(): + vars_cont += 1 + else: + vars_int += 1 + nonants_total = nonants_int = 0 + for node in getattr(model, "_mpisppy_node_list", []): + for v in node.nonant_vardata_list: + nonants_total += 1 + if not v.is_continuous(): + nonants_int += 1 + return {"vars_int": vars_int, "vars_cont": vars_cont, + "nonants_total": nonants_total, "nonants_int": nonants_int} + + +def _build_probe_scenario(module, cfg): + """Instantiate a single RAW scenario (no bundle/ADMM/cvar wrapper) from the + model module so OOTB can measure per-scenario size. Raises if the model + cannot build.""" + names = module.scenario_names_creator(1) + kwargs = module.kw_creator(cfg) + return module.scenario_creator(names[0], **kwargs) + + +def verify_instantiation(module, cfg) -> dict: + """Build a single scenario to confirm the model instantiates -- a cheap + model smoke-test -- and return its size profile. SHARED CODE: used by the + base/plus probe in gather_facts and by --inspect-only when given WITHOUT + --out-of-the-box (the standalone model smoke-test).""" + model = _build_probe_scenario(module, cfg) + return _size_profile(model) -def verify_instantiation(module, cfg): - """Build a single scenario to confirm the model instantiates -- a cheap - model smoke-test. SHARED CODE: used by --inspect-only when given WITHOUT - --out-of-the-box, and by the base/plus probe in gather_facts (which also - reads the size profile off the built model). STUB: wiring deferred.""" - # TODO(wiring): name = module.scenario_names_creator(1, cfg=cfg)[0] - # model = module.scenario_creator(name, **kwargs) # raises if it can't build - # return the size profile (vars_int, vars_cont, nonants_total, nonants_int) - raise NotImplementedError("scenario instantiation/verification not wired yet") - - -def gather_facts(module, cfg) -> Facts: - """Assemble Facts from the environment, the model module, and cfg. - STUB: the pieces below marked TODO are the integration work.""" - policy = load_policy() +def gather_facts(module, cfg, effort: str, policy: dict) -> Facts: + """Assemble Facts from the environment, the model module, and cfg. The + `effort` tier sets how deep we look: minus reads structure only; base/plus + also instantiate `probe_scenarios` scenario(s) for the size profile.""" solvers = _detect_available_solvers(policy["solver"]["preference_order"]) + bf = cfg.get("branching_factors") facts = Facts( - module_name=cfg.get("module_name", ""), + module_name=cfg.get("module_name", "") or "", num_ranks=_inspect_ranks(cfg), available_solvers=solvers, num_scens=_detect_num_scens(module, cfg), + effort=effort, + multistage=bf is not None, + branching_factors=list(bf) if bf is not None else None, + user_solver_name=cfg.get("solver_name") or cfg.get("EF_solver_name"), num_cores=os.cpu_count(), under_slurm=("SLURM_JOB_ID" in os.environ), - # TODO(wiring): memory_gb (OS/SLURM-dependent), multistage flag, - # user_flags (which options the user explicitly set on cfg), and -- at - # the base/plus tiers -- the probe size profile (vars_int, vars_cont, - # nonants_total, nonants_int) by instantiating probe_scenarios scenarios. - user_flags=set(), + user_flags=_user_flags(), ) + if effort in ("base", "plus"): + # one probe scenario (discarded) feeds the size-aware decisions; the + # real, layout-correct instantiation happens later in the driver. + profile = verify_instantiation(module, cfg) + facts.vars_int = profile["vars_int"] + facts.vars_cont = profile["vars_cont"] + facts.nonants_total = profile["nonants_total"] + facts.nonants_int = profile["nonants_int"] + # plus tier (PR2): instantiate all + brief timed solve for solve-time facts. return facts -def out_of_the_box(module, cfg): - """Top-level entry (STUB). Probe, recommend, report the config up front, - run, then print Suggestions afterward.""" - facts = gather_facts(module, cfg) - policy = load_policy() +def apply_decision(decision: Decision, cfg) -> None: + """Apply the recommended args onto the Config so the normal driver path runs + the chosen configuration. User-set flags are already absent from + decision.args (recommend() never adds them), so this never overrides the + user (requirement 0). Pyomo coerces string values through each option's + domain.""" + if decision.run_ef: + cfg["EF"] = True + for a in decision.args: + key = a.flag[2:].replace("-", "_") # "--solver-name" -> "solver_name" + cfg[key] = True if a.value is None else a.value + + # The EF path reads cfg.EF_solver_name, the decomposition path cfg.solver_name + # -- two different keys. A naive user (and OOTB itself) naturally sets + # --solver-name; if OOTB then runs the EF, carry that name over so the EF + # path is never left without a solver. (recommend() already emits + # --EF-solver-name; this guarantees it independent of how cfg was populated.) + if decision.run_ef and cfg.get("EF_solver_name") is None \ + and cfg.get("solver_name") is not None: + cfg["EF_solver_name"] = cfg["solver_name"] + + +@dataclass +class OOTBState: + decision: Decision + facts: Facts + policy: dict + effort: str + + +def configure(module, cfg) -> OOTBState: + """Top-level OOTB entry called by the driver right after arg parsing. Probe + the environment/model, recommend a configuration, print it and the + equivalent command line (rank 0), and APPLY it onto cfg so the normal driver + path runs it. Returns the state so the driver can print Suggestions after + the run (see report_suggestions).""" + effort, policy_path = effort_and_policy(cfg) + policy = load_policy(policy_path) + facts = gather_facts(module, cfg, effort, policy) decision = recommend(facts, policy) - # report the configuration up front so the user sees what is running - # (rank-0 printing is the caller's concern). - print(f"[out-of-the-box] using policy {policy['policy_version']}") - for note in decision.notes: - print(f" - {note}") - print(f"[out-of-the-box] equivalent command line:\n " - f"{decision.command_line(facts.num_ranks)}") - - if cfg.get("inspect_only", None) is not None: - # --inspect-only: inspection is already done -- instantiation, and any - # plus calibration solves, happened in gather_facts (resolution B) -- - # so just skip the production run. facts.num_ranks may be an assumed - # count (--inspect-only N) for HPC planning. - print(f"[out-of-the-box] --inspect-only: planning for {facts.num_ranks} " - f"ranks; skipping the production run") - outcome = None - else: - # TODO(wiring): apply decision.args onto cfg, then run via the normal - # generic_cylinders driver path (EF vs cylinders); capture an `outcome` - # (convergence, gap, iters, time) to enrich the suggestions. - outcome = None - - # Suggestions are written AFTER the run (req. 4), labelled "Suggestions". - decision.suggestions = make_suggestions(decision, facts, policy, outcome) - if decision.suggestions: + if _rank0(): + print(f"[out-of-the-box] tier '{effort}', policy " + f"{policy.get('policy_version', '?')}") + for note in decision.notes: + print(f" - {note}") + print("[out-of-the-box] equivalent command line:\n " + + decision.command_line(facts)) + + apply_decision(decision, cfg) + return OOTBState(decision=decision, facts=facts, policy=policy, effort=effort) + + +def report_suggestions(state: OOTBState, outcome: dict | None = None) -> None: + """Print the prioritized "Suggestions" list (req. 4), AFTER the run so it can + reflect how the run went (via `outcome`). Rank 0 only.""" + if not _rank0(): + return + state.decision.suggestions = make_suggestions( + state.decision, state.facts, state.policy, outcome) + if state.decision.suggestions: print("[out-of-the-box] Suggestions:") - for s in decision.suggestions: + for s in state.decision.suggestions: print(f" * {s}") - return decision + +def inspect_only_standalone(module, cfg) -> None: + """--inspect-only WITHOUT any --out-of-the-box tier: verify one scenario can + be instantiated (a cheap model smoke-test), report its size, and stop. Rank + 0 prints; all ranks build (cheap, deterministic).""" + profile = verify_instantiation(module, cfg) + if _rank0(): + print("[inspect-only] model instantiates; one-scenario size profile:") + for k, v in profile.items(): + print(f" {k}: {v}") + print("[inspect-only] no out-of-the-box tier given; " + "skipping the production run.") diff --git a/mpisppy/generic/parsing.py b/mpisppy/generic/parsing.py index 85e85cf67..951fc99e2 100644 --- a/mpisppy/generic/parsing.py +++ b/mpisppy/generic/parsing.py @@ -172,6 +172,7 @@ def parse_args(m): # TBD - think about adding directory for json options files cfg.mmw_args() + cfg.ootb_args() from mpisppy.generic.admm import admm_args admm_args(cfg) diff --git a/mpisppy/generic_cylinders.py b/mpisppy/generic_cylinders.py index d8d8ab832..9ee1a34d4 100644 --- a/mpisppy/generic_cylinders.py +++ b/mpisppy/generic_cylinders.py @@ -45,6 +45,21 @@ if hasattr(module, "get_mpisppy_helper_object"): module = module.get_mpisppy_helper_object(cfg) + # Out-of-the-box auto-configuration (and the OOTB-independent --inspect-only + # dry run). configure() probes the environment + model, prints the chosen + # configuration and equivalent command line, and mutates cfg so the normal + # driver path below runs it. --inspect-only stops before the production run. + from mpisppy.generic import out_of_the_box as ootb + ootb_state = None + if ootb.requested(cfg): + ootb_state = ootb.configure(module, cfg) + if cfg.get("inspect_only") is not None: + ootb.report_suggestions(ootb_state) # config-time suggestions only + sys.exit(0) + elif cfg.get("inspect_only") is not None: + ootb.inspect_only_standalone(module, cfg) + sys.exit(0) + bundle_wrapper = None # the default if proper_bundles(cfg): # Nonant name validation will fail with proper bundles because @@ -145,3 +160,8 @@ def scenario_denouement(rank, sname, s): scenario_denouement, bundle_wrapper=bundle_wrapper) if mmw_requested(cfg): do_mmw(fname, cfg, wheel=wheel) + + # Out-of-the-box: the prioritized "Suggestions" list is printed AFTER the + # run so it can also reflect how the run went (req. 4). + if ootb_state is not None: + ootb.report_suggestions(ootb_state) diff --git a/mpisppy/utils/config.py b/mpisppy/utils/config.py index d49dc383b..e5945825f 100644 --- a/mpisppy/utils/config.py +++ b/mpisppy/utils/config.py @@ -1671,6 +1671,64 @@ def mmw_args(self): default=None, ) + def ootb_args(self): + """Out-of-the-box (OOTB) auto-configuration flags. + + Three mutually-exclusive effort tiers plus the (OOTB-independent) + ``--inspect-only`` dry run. See doc/designs/out_of_the_box_design.md + and mpisppy/generic/out_of_the_box.py. + + Each tier flag takes an OPTIONAL value: the path to a policy file. + Declared ``domain=str, default=None`` with ``nargs='?', const=''`` so + there are three states -- absent (None, OOTB off), bare flag ('', use + the shipped default policy), and ``--out-of-the-box PATH`` (that policy + file). A bool domain cannot be used because pyomo forces ``store_true`` + (which takes no value); ``str`` + ``nargs='?'`` is the supported + optional-value form. Detection keys on ``is not None`` (not truthiness) + because the bare-flag value is the empty string. + """ + self.add_to_config( + "out_of_the_box", + description="Auto-configure a defensible run from the environment " + "and model (base tier: one probe scenario). Optional value is a " + "policy-file path; bare flag uses the shipped default policy. " + "User-supplied options always win.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "out_of_the_box_minus", + description="Like --out-of-the-box but instantiates nothing " + "(structural decisions only; cannot bundle). Optional policy-file " + "path; bare flag uses the default policy.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "out_of_the_box_plus", + description="Like --out-of-the-box but instantiates all scenarios " + "and does a brief timed solve for more information (NOT a tuning " + "tool). Optional policy-file path; bare flag uses the default " + "policy.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "inspect_only", + description="Do the inspection (and, with --out-of-the-box-plus, " + "the brief calibration solve), print the configuration, the " + "equivalent command line, and config-time suggestions, then STOP " + "before the production run. Optional value is an assumed MPI rank " + "count for HPC planning (e.g. --inspect-only 512); bare flag uses " + "the actually-detected rank count.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": "detected"}, + ) + #================ def create_parser(self,progname=None): # seldom used From 692162613c30446365f29fb50eff01213bd13652 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 17:17:16 -0700 Subject: [PATCH 29/34] Out-of-the-box: policy-file validator (design sec. 8) Add mpisppy/generic/ootb_validate.py -- a runnable validator that checks a policy file is well-formed and that its recommended configurations make sense, and (on demand) that they actually run, using the mpi-sppy examples as test models. Three layers: 1. STATIC (schema): required keys/types; every referenced flag is a real CLI option; spoke rungs, rho setters, _cold_start_guess keys, and DECOMPOSITION_FLAGS all line up with the real generic_cylinders vocabulary. 2. DECISION (recommend() only, no solves): synthetic hand-built Facts (the CI-gating subset) asserting EF-when-it-should, decompose-when-it-should, forced-decomp-wins, bundling validity, and no conflicting rho setters; with --examples, also probe-instantiated real models (farmer / sizes / aircond). 3. RUN (--run; slow, needs a solver + mpiexec, never a CI gate): actually run the recommended EF and forced-decomposition configs and FLAG (a) an EF that misses a 1% gap in ten minutes and (b) cylinders that max out on iterations. Everything else (objective, gap, iters, walltime) is recorded for a human. Produces a human-readable + machine-readable (--json) report naming the policy file and policy_version. CI gate: mpisppy/tests/test_ootb_validate.py runs only layers 1 + 2-synthetic on every shipped policy (solver-free, ~0.5s), plus negative tests proving the validator actually fails on malformed policies. Wired into run_coverage.bash and test_pr_and_main.yml in the same commit. Supporting changes: - parsing.add_driver_args(cfg, m=None): the arg declaration split out of parse_args so the validator can get the authoritative valid-flag set without parsing a command line (m=None skips the model-specific adder). - ootb_validate scrubs OMPI_/PMI_/... from child mpiexec environments: importing mpi-sppy makes the validator a singleton MPI process, and leaving those vars in the child made a fresh mpiexec abort instantly. - Policy fix the validator itself caught: spoke_ladder._cold_start_guess listed prose ("ladder ordering of priorities 3-6"), not a real key; moved that note into the _comment so _cold_start_guess names only real keys. Verified: layers 1+2 PASS on the shipped policy (50 static + 8 decision); --examples PASS across all three models; --run completes all six runs (farmer / sizes / aircond, EF + forced-decompose) with none flagged. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/test_pr_and_main.yml | 4 + .../ootb_policies/ootb_policy_2026-06-28.json | 4 +- mpisppy/generic/ootb_validate.py | 688 ++++++++++++++++++ mpisppy/generic/parsing.py | 26 +- mpisppy/tests/test_ootb_validate.py | 116 +++ run_coverage.bash | 3 + 6 files changed, 832 insertions(+), 9 deletions(-) create mode 100644 mpisppy/generic/ootb_validate.py create mode 100644 mpisppy/tests/test_ootb_validate.py diff --git a/.github/workflows/test_pr_and_main.yml b/.github/workflows/test_pr_and_main.yml index 7d5532770..6e0221135 100644 --- a/.github/workflows/test_pr_and_main.yml +++ b/.github/workflows/test_pr_and_main.yml @@ -124,6 +124,10 @@ jobs: run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_generic_cylinders.py -v + - name: Test out-of-the-box policy validator (layers 1 + 2-synthetic) + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_validate.py -v + - name: Test xhat from file run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_xhat_from_file.py -v diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 26c99c089..b7b035cbf 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -47,7 +47,7 @@ }, "spoke_ladder": { - "_comment": "Ordered list of spokes to switch on as ranks allow, lowest priority number first. Only flags WIRED in mpisppy/generic/spokes.py appear here. The 3-rank minimum fills priorities 1 and 2 (one outer + one inner bound) so there is always an optimality gap. Each rung consumes one cylinder; see rank_allocation for how leftover ranks are spent.", + "_comment": "Ordered list of spokes to switch on as ranks allow, lowest priority number first. Only flags WIRED in mpisppy/generic/spokes.py appear here. The 3-rank minimum fills priorities 1 and 2 (one outer + one inner bound) so there is always an optimality gap. Each rung consumes one cylinder; see rank_allocation for how leftover ranks are spent. NOTE: the ORDERING of priorities 3-6 is itself a cold-start guess (no benchmark data yet); _cold_start_guess lists only real keys, so it cannot name the ordering, but treat the priority order below as provisional.", "rungs": [ {"priority": 1, "flag": "--lagrangian", "factory": "lagrangian_spoke", "bound": "outer", "note": "Lagrangian dual bound; cheap and robust. Core."}, {"priority": 2, "flag": "--xhatshuffle", "factory": "xhatshuffle_spoke", "bound": "inner", "note": "Incumbent finder over shuffled scenarios. Core."}, @@ -62,7 +62,7 @@ "inner": 1 }, "max_cylinders": 7, - "_cold_start_guess": ["max_cylinders", "ladder ordering of priorities 3-6"] + "_cold_start_guess": ["max_cylinders"] }, "rank_allocation": { diff --git a/mpisppy/generic/ootb_validate.py b/mpisppy/generic/ootb_validate.py new file mode 100644 index 000000000..51b59de75 --- /dev/null +++ b/mpisppy/generic/ootb_validate.py @@ -0,0 +1,688 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) policy-file validator. + +Given a dated policy file, check it is well-formed and that its recommended +configurations make sense -- and, on demand, that they actually RUN -- using the +mpi-sppy examples as test models. See doc/designs/out_of_the_box_design.md (sec. +8). It does NOT assert "expected results" (there is no cheap oracle); instead it +records what happened and FLAGS two run failure modes for a human to review. + + python -m mpisppy.generic.ootb_validate # layers 1+2 + python -m mpisppy.generic.ootb_validate --examples # +real models + python -m mpisppy.generic.ootb_validate --run # +layer 3 runs + +Three layers, fast to slow: + + 1. STATIC (schema): JSON parses; required keys/types; every referenced flag is + a real CLI option; spoke rungs / rho setters / DECOMPOSITION_FLAGS all line + up with the actual generic_cylinders vocabulary. + 2. DECISION (recommend() only, no solves): on hand-built synthetic Facts (the + CI-gating subset) and, with --examples, on probe-instantiated real models. + Asserts EF-when-it-should, decompose-when-it-should, forced-decomp-wins, + bundling validity, and no conflicting rho setters. + 3. RUN (--run; slow, needs a solver, NEVER a CI gate): actually run the + recommended configs on the small examples and FLAG (a) an EF that misses a + 1% gap in ten minutes and (b) cylinders that max out on iterations. + +The CI gate (mpisppy/tests/test_ootb_validate.py) runs only layers 1 + +2-synthetic on the shipped policy file(s) -- solver-free, fast. +""" + +from __future__ import annotations + +import argparse +import importlib +import json +import os +import re +import subprocess +import sys +import time +from dataclasses import dataclass + +import mpisppy.utils.config as config +from mpisppy.generic import parsing +from mpisppy.generic import out_of_the_box as ootb + + +# Validator settings (NOT policy): the two layer-3 auto-flag thresholds. +EF_GAP_TARGET = 0.01 # flag an EF that misses this MIP gap ... +EF_TIME_LIMIT_SEC = 600 # ... within this wall-clock budget (ten minutes) + + +# --------------------------------------------------------------------------- +# Result carriers +# --------------------------------------------------------------------------- + + +@dataclass +class Check: + layer: str # "static" | "decision" + name: str + ok: bool + detail: str = "" + + def as_dict(self) -> dict: + return {"layer": self.layer, "name": self.name, "ok": self.ok, + "detail": self.detail} + + +@dataclass +class RunRecord: + example: str + env: dict + mode: str # "EF" | "decompose" + returncode: int | None = None + walltime: float | None = None + objective: float | None = None + rel_gap: float | None = None + iterations: int | None = None + flagged: bool = False + flag_reason: str = "" + detail: str = "" + + def as_dict(self) -> dict: + return {k: getattr(self, k) for k in + ("example", "env", "mode", "returncode", "walltime", "objective", + "rel_gap", "iterations", "flagged", "flag_reason", "detail")} + + +# --------------------------------------------------------------------------- +# The example models the validator exercises (layers 2-examples and 3). +# --------------------------------------------------------------------------- + + +def _repo_root() -> str: + # mpisppy/generic/ootb_validate.py -> repo root is three directories up. + return os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + + +def example_models() -> list[dict]: + root = _repo_root() + return [ + {"name": "farmer", "kind": "2-stage continuous", + "dir": os.path.join(root, "examples", "farmer"), "module": "farmer", + "scens": {"num_scens": 6}}, + {"name": "sizes", "kind": "2-stage MIP", + "dir": os.path.join(root, "examples", "sizes"), "module": "sizes", + "scens": {"num_scens": 3}}, + {"name": "aircond", "kind": "multistage", + "dir": os.path.join(root, "mpisppy", "tests", "examples"), + "module": "aircond", + "scens": {"branching_factors": [3, 2]}}, + ] + + +def _scen_cli(spec: dict) -> list: + """The --num-scens / --branching-factors CLI args for a spec. Branching + factors are passed as ONE space-joined token (the ListOf(int) domain parses + a single string, so '--branching-factors 3 2' would leave '2' unparsed).""" + s = spec["scens"] + if "num_scens" in s: + return ["--num-scens", str(s["num_scens"])] + return ["--branching-factors", " ".join(str(b) for b in s["branching_factors"])] + + +def _child_env() -> dict: + """Environment for subprocess runs with this (singleton-MPI) process's MPI + variables scrubbed. Importing mpi-sppy initializes mpi4py.MPI here, so the + parent IS an MPI process; leaving OMPI_/PMI_/... in the child's environment + makes a fresh mpiexec think it is being relaunched inside a rank and fail.""" + return {k: v for k, v in os.environ.items() + if not k.startswith(("OMPI_", "PMI_", "PMIX_", "MPIR_", "HYDRA_"))} + + +# --------------------------------------------------------------------------- +# Shared helpers +# --------------------------------------------------------------------------- + + +def valid_flags() -> set: + """The authoritative set of real CLI flags, from the same declaration the + driver uses (parsing.add_driver_args with no model module).""" + ref = config.Config() + parsing.add_driver_args(ref) + return {"--" + name.replace("_", "-") for name in ref} + + +def _is_number(x) -> bool: + return isinstance(x, (int, float)) and not isinstance(x, bool) + + +# --------------------------------------------------------------------------- +# Layer 1: static (schema) checks +# --------------------------------------------------------------------------- + + +def validate_static(policy: dict) -> list: + checks = [] + flags = valid_flags() + + def add(name, ok, detail=""): + checks.append(Check("static", name, ok, detail)) + + # required top-level keys + required = ["schema_version", "policy_version", "ef_fallback", "solver", + "hub", "spoke_ladder", "rank_allocation", "effort_scaling", + "bundle_sizing", "option_categories", "additional_options", + "suggestions"] + missing = [k for k in required if k not in policy] + add("required top-level keys present", not missing, + f"missing: {missing}" if missing else "all present") + if missing: + return checks # the rest assume the structure exists + + # ef_fallback numbers + ef = policy["ef_fallback"] + add("ef_fallback.min_ranks_for_decomposition is a positive int", + isinstance(ef.get("min_ranks_for_decomposition"), int) + and ef["min_ranks_for_decomposition"] >= 1, + str(ef.get("min_ranks_for_decomposition"))) + for k in ("ef_if_num_scens_at_most", "ef_effort_budget", "ef_target_seconds"): + add(f"ef_fallback.{k} is a positive number", + _is_number(ef.get(k)) and ef[k] > 0, str(ef.get(k))) + + # solver + sp = policy["solver"] + pref = sp.get("preference_order") + add("solver.preference_order is a non-empty list of strings", + isinstance(pref, list) and len(pref) > 0 + and all(isinstance(s, str) for s in pref)) + for key in ("commercial", "qp_capable", "lp_mip_only_force_linearize_prox"): + val = sp.get(key, []) + add(f"solver.{key} is a subset of preference_order", + isinstance(val, list) and set(val) <= set(pref or []), + f"stray: {sorted(set(val) - set(pref or []))}") + + # spoke ladder + ladder = policy["spoke_ladder"] + rungs = ladder.get("rungs", []) + rung_flags = [r.get("flag") for r in rungs] + add("spoke_ladder.rungs flags are real CLI options", + all(f in flags for f in rung_flags), + f"unknown: {[f for f in rung_flags if f not in flags]}") + add("spoke_ladder.rungs flags are decomposition (spoke) flags", + all(f in ootb.DECOMPOSITION_FLAGS for f in rung_flags), + f"not spokes: {[f for f in rung_flags if f not in ootb.DECOMPOSITION_FLAGS]}") + add("spoke_ladder.rungs have unique priorities", + len({r.get("priority") for r in rungs}) == len(rungs)) + add("spoke_ladder.rungs bounds are outer/inner", + all(r.get("bound") in ("outer", "inner") for r in rungs)) + core = ladder.get("core_roster_min", {}) + add("spoke_ladder.core_roster_min has outer & inner counts", + isinstance(core.get("outer"), int) and isinstance(core.get("inner"), int)) + add("spoke_ladder.max_cylinders is an int >= core roster size", + isinstance(ladder.get("max_cylinders"), int) + and ladder["max_cylinders"] >= 1 + core.get("outer", 0) + core.get("inner", 0)) + # the ladder must be able to satisfy the core roster + avail = {"outer": 0, "inner": 0} + for r in rungs: + if r.get("bound") in avail: + avail[r["bound"]] += 1 + need = {k: core.get(k, 0) for k in ("outer", "inner")} + add("spoke_ladder can satisfy core_roster_min", + avail["outer"] >= need["outer"] and avail["inner"] >= need["inner"], + f"available {avail} vs needed {need}") + + # rank allocation + ra = policy["rank_allocation"] + add("rank_allocation.min_ranks_per_cylinder is a positive int", + isinstance(ra.get("min_ranks_per_cylinder"), int) + and ra["min_ranks_per_cylinder"] >= 1) + add("rank_allocation.default_rank_ratio is a positive number", + _is_number(ra.get("default_rank_ratio")) and ra["default_rank_ratio"] > 0) + rr = ra.get("rank_ratios", {}) + add("rank_allocation.rank_ratios keys are real spoke flags", + all(f in flags for f in rr), f"unknown: {[f for f in rr if f not in flags]}") + add("rank_allocation.rank_ratios values are positive numbers", + all(_is_number(v) and v > 0 for v in rr.values())) + + # effort scaling + es = policy["effort_scaling"] + for k in ("cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff"): + add(f"effort_scaling.{k} is a non-negative number", + _is_number(es.get(k)) and es[k] >= 0, str(es.get(k))) + add("effort_scaling.int_exponent >= 1 (integers at least linear)", + _is_number(es.get("int_exponent")) and es["int_exponent"] >= 1) + + # bundle sizing + bs = policy["bundle_sizing"] + for k in ("min_scens_to_consider_bundling", "min_bundles_per_intra_rank", + "base_max_hardness_vs_single_scenario", + "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"): + add(f"bundle_sizing.{k} is a positive number", + _is_number(bs.get(k)) and bs[k] > 0, str(bs.get(k))) + add("bundle_sizing.base_max_hardness_vs_single_scenario >= 1", + _is_number(bs.get("base_max_hardness_vs_single_scenario")) + and bs["base_max_hardness_vs_single_scenario"] >= 1) + + # option categories + oc = policy.get("option_categories", {}) + for name, cat in oc.items(): + if name.startswith("_"): + continue + add(f"option_categories.{name}.flag is a real CLI option", + cat.get("flag") in flags, str(cat.get("flag"))) + sup = cat.get("superseded_by", []) + add(f"option_categories.{name}.superseded_by are real CLI options", + isinstance(sup, list) and all(f in flags for f in sup), + f"unknown: {[f for f in sup if f not in flags]}") + # the rho_setter concern, if present, must list all rho setters (only one may + # be active, so OOTB stacking a second would be a hard error). + if "rho_setter" in oc: + rho_setters = {"--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho"} + sup = set(oc["rho_setter"].get("superseded_by", [])) + add("option_categories.rho_setter.superseded_by lists every rho setter", + rho_setters <= sup, f"missing: {sorted(rho_setters - sup)}") + + # additional options catch-all + ao = policy.get("additional_options", {}).get("options", []) + for opt in ao: + add(f"additional_options flag {opt.get('flag')} is a real CLI option", + opt.get("flag") in flags, str(opt.get("flag"))) + sup = opt.get("superseded_by", [opt.get("flag")]) + add(f"additional_options {opt.get('flag')} superseded_by are real options", + all(f in flags for f in sup)) + + # suggestions: disabled names real generators + gen_names = {g.__name__ for g in ootb.SUGGESTION_GENERATORS} + disabled = policy.get("suggestions", {}).get("disabled", []) + add("suggestions.disabled names real generators", + isinstance(disabled, list) and all(d in gen_names for d in disabled), + f"unknown: {[d for d in disabled if d not in gen_names]}") + + # _cold_start_guess entries name real keys in their block + for block_name, block in policy.items(): + if not isinstance(block, dict) or "_cold_start_guess" not in block: + continue + guesses = block["_cold_start_guess"] + bad = [g for g in guesses if g not in block] + add(f"{block_name}._cold_start_guess entries name real keys", not bad, + f"stray: {bad}") + + # DECOMPOSITION_FLAGS line up with the real vocabulary + bad = sorted(f for f in ootb.DECOMPOSITION_FLAGS if f not in flags) + add("out_of_the_box.DECOMPOSITION_FLAGS are all real CLI options", not bad, + f"unknown: {bad}") + + return checks + + +# --------------------------------------------------------------------------- +# Layer 2: decision checks on hand-built synthetic Facts (the CI-gating subset) +# --------------------------------------------------------------------------- + + +def _facts(**kw): + """Build a Facts with sensible defaults for the synthetic decision checks.""" + base = dict(module_name="synthetic", num_ranks=3, + available_solvers={"gurobi_persistent"}, num_scens=10, + effort="base", vars_int=0, vars_cont=10, nonants_total=3, + nonants_int=0) + base.update(kw) + return ootb.Facts(**base) + + +def _bundle_arg(decision): + for a in decision.args: + if a.flag == "--scenarios-per-bundle": + return int(a.value) + return None + + +def _rho_setter_args(decision): + setters = {"--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho"} + return [a.flag for a in decision.args if a.flag in setters] + + +def validate_decisions_synthetic(policy: dict) -> list: + checks = [] + minr = policy["ef_fallback"]["min_ranks_for_decomposition"] + + def add(name, ok, detail=""): + checks.append(Check("decision", name, ok, detail)) + + # EF when below the rank floor + d = ootb.recommend(_facts(num_ranks=minr - 1), policy) + add("below rank floor -> EF", d.run_ef and d.ef_reason == "min_ranks", + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # EF when the whole problem is tiny (base effort gate) + d = ootb.recommend(_facts(num_ranks=minr, num_scens=2, + vars_cont=1, vars_int=0), policy) + add("tiny problem above rank floor -> EF", d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # NOT EF when the whole problem is large / integer-heavy + d = ootb.recommend(_facts(num_ranks=6, num_scens=1000, + vars_cont=500, vars_int=200, nonants_int=50), policy) + add("large integer-heavy problem -> decompose (not EF)", not d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # forced decomposition wins even on a tiny problem + d = ootb.recommend(_facts(num_ranks=minr, num_scens=2, vars_cont=1, + user_flags={"--lagrangian", "--xhatshuffle"}), policy) + add("user-forced decomposition (>= rank floor) -> never EF", not d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # bundling validity: when OOTB bundles, spb divides num_scens & #bundles>=ranks + d = ootb.recommend(_facts(num_ranks=minr, num_scens=120, vars_cont=5, + user_flags={"--lagrangian"}), policy) + spb = _bundle_arg(d) + if spb is None: + add("bundling validity (no bundle chosen here)", True, "no --scenarios-per-bundle") + else: + ok = (120 % spb == 0) and (120 // spb >= d.intra_ranks) + add("bundling: spb divides num_scens and #bundles >= intra_ranks", ok, + f"spb={spb}, #bundles={120 // spb}, intra_ranks={d.intra_ranks}") + + # minus tier never bundles (no size profile) + d = ootb.recommend(_facts(effort="minus", vars_int=None, vars_cont=None, + nonants_total=None, nonants_int=None, + num_ranks=minr, num_scens=120, + user_flags={"--lagrangian"}), policy) + add("minus tier never bundles", _bundle_arg(d) is None, + f"spb={_bundle_arg(d)}") + + # no conflicting rho setters: the user's rho setter must suppress OOTB's + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_int=5, + vars_cont=10, nonants_int=2, + user_flags={"--lagrangian", "--sensi-rho"}), policy) + add("user rho setter suppresses OOTB's (<=0 OOTB rho setters)", + len(_rho_setter_args(d)) == 0, f"OOTB rho setters: {_rho_setter_args(d)}") + + # OOTB adds at most one rho setter on its own + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_int=5, vars_cont=10, + user_flags={"--lagrangian"}), policy) + add("OOTB adds at most one rho setter", len(_rho_setter_args(d)) <= 1, + f"OOTB rho setters: {_rho_setter_args(d)}") + + return checks + + +# --------------------------------------------------------------------------- +# Layer 2: decision checks on real, probe-instantiated example models (out of CI) +# --------------------------------------------------------------------------- + + +def _load_example_module(spec: dict): + if spec["dir"] not in sys.path: + sys.path.insert(0, spec["dir"]) + return importlib.import_module(spec["module"]) + + +def _example_cfg(spec: dict, module): + """Build a fully-declared cfg for an example without parsing a command line, + then set its scenario count / branching factors from spec["scen_args"].""" + cfg = config.Config() + parsing.add_driver_args(cfg, module) + s = spec["scens"] + if "num_scens" in s: + cfg.num_scens = s["num_scens"] + else: + cfg.branching_factors = list(s["branching_factors"]) + return cfg + + +def validate_decisions_examples(policy: dict) -> list: + checks = [] + for spec in example_models(): + name = spec["name"] + try: + module = _load_example_module(spec) + cfg = _example_cfg(spec, module) + except Exception as e: # noqa: BLE001 + checks.append(Check("decision", f"example {name}: set up", False, + f"{type(e).__name__}: {e}")) + continue + for nranks in (1, 3, 8): + try: + cfg.inspect_only = str(nranks) # plan as if nranks (no MPI) + facts = ootb.gather_facts(module, cfg, "base", policy) + d = ootb.recommend(facts, policy) + except Exception as e: # noqa: BLE001 + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: recommend", + False, f"{type(e).__name__}: {e}")) + continue + # the probe must have populated a size profile + prof_ok = facts.vars_cont is not None and facts.vars_int is not None + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: probe size profile", + prof_ok, + f"int={facts.vars_int} cont={facts.vars_cont} " + f"nonants={facts.nonants_total}")) + # internal consistency of the recommendation + spb = _bundle_arg(d) + bundle_ok = spb is None or (facts.num_scens % spb == 0 + and facts.num_scens // spb >= d.intra_ranks) + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: bundling valid", + bundle_ok, f"spb={spb} intra={d.intra_ranks} " + f"num_scens={facts.num_scens}")) + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: <=1 rho setter", + len(_rho_setter_args(d)) <= 1, + str(_rho_setter_args(d)))) + return checks + + +# --------------------------------------------------------------------------- +# Layer 3: actually run the recommended configs (--run; slow; never a CI gate) +# --------------------------------------------------------------------------- + + +_OBJ_RE = re.compile(r"EF objective:\s*([-\d.eE+]+)") +# A termination stats row, tolerating global_toc's leading "[ 0.86] " prefix: +# [ 0.86] 99 X -124309.6133 -123177.9852 0.910% 1131.6281 +_STAT_RE = re.compile(r"^(?:\[[^\]]*\]\s*)?(\d+)\s+.*?([\d.]+)%\s+[-\d.eE+]+\s*$") + + +def _parse_decompose(out: str): + """Return (converged, iterations, rel_gap) parsed from a decomposition run's + stdout. 'Terminating based on inter-cylinder' (or 'Cylinder convergence') + marks gap convergence; otherwise the run exhausted its iterations.""" + converged = ("Terminating based on inter-cylinder" in out + or "Cylinder convergence" in out) + iters = rel_gap = None + seen_stats = False + for line in out.splitlines(): + if "Statistics at termination" in line: + seen_stats = True + continue + if seen_stats: + m = _STAT_RE.match(line) + if m: + iters = int(m.group(1)) + rel_gap = float(m.group(2)) / 100.0 + return converged, iters, rel_gap + + +def _run_one(spec, mode, nranks, extra_args, timeout): + """Run one example via generic_cylinders in a subprocess and time it.""" + if mode == "EF": + cmd = [sys.executable, "-m", "mpisppy.generic_cylinders"] + else: + cmd = ["mpiexec", "-np", str(nranks), sys.executable, "-m", "mpi4py", + "-m", "mpisppy.generic_cylinders"] + cmd += ["--module-name", spec["module"]] + _scen_cli(spec) + extra_args + rec = RunRecord(example=spec["name"], + env={"ranks": nranks, "mode": mode}, mode=mode) + start = time.time() + try: + proc = subprocess.run(cmd, cwd=spec["dir"], capture_output=True, + text=True, timeout=timeout, env=_child_env()) + rec.returncode = proc.returncode + out = proc.stdout + "\n" + proc.stderr + rec.detail = (proc.stderr.strip().splitlines() or [""])[-1] + except subprocess.TimeoutExpired: + rec.returncode = None + rec.walltime = time.time() - start + rec.flagged = True + rec.flag_reason = f"timed out after {timeout}s" + return rec + rec.walltime = time.time() - start + return rec, out + + +def validate_runs(policy_path: str, *, ef_time_limit=EF_TIME_LIMIT_SEC, + ef_gap=EF_GAP_TARGET) -> list: + """Run two configurations per example and flag the two failure modes.""" + records = [] + pol_arg = ["--out-of-the-box", policy_path] if policy_path else ["--out-of-the-box"] + for spec in example_models(): + # (a) EF: few ranks -> OOTB picks the EF. Flag if it misses the gap/time. + res = _run_one(spec, "EF", 1, ["--out-of-the-box-minus"] + ( + ["--EF-mipgap", str(ef_gap)] if spec["kind"] == "2-stage MIP" else []), + timeout=ef_time_limit) + if isinstance(res, RunRecord): # timed out + res.flag_reason = f"EF missed a {ef_gap:.0%} gap within {ef_time_limit}s" + records.append(res) + else: + rec, out = res + m = _OBJ_RE.search(out) + rec.objective = float(m.group(1)) if m else None + if rec.returncode != 0 or rec.objective is None: + rec.flagged = True + rec.flag_reason = "EF run did not complete cleanly" + records.append(rec) + + # (b) decomposition (forced via --lagrangian, to exercise that path even + # on a small problem). Flag if it maxes out iterations instead of + # converging on the inter-cylinder gap. + res = _run_one(spec, "decompose", 3, pol_arg + ["--lagrangian"], + timeout=ef_time_limit) + if isinstance(res, RunRecord): # timed out + records.append(res) + else: + rec, out = res + converged, iters, rel_gap = _parse_decompose(out) + rec.iterations, rec.rel_gap = iters, rel_gap + if rec.returncode == 0 and not converged: + rec.flagged = True + rec.flag_reason = "cylinders maxed out on iterations (no gap convergence)" + elif rec.returncode != 0: + rec.flagged = True + rec.flag_reason = "decomposition run did not complete cleanly" + records.append(rec) + return records + + +# --------------------------------------------------------------------------- +# Orchestration + report +# --------------------------------------------------------------------------- + + +def run_validation(policy_path: str, *, examples=False, run=False) -> dict: + """Run the requested layers and return a machine-readable report dict.""" + report = {"policy_file": policy_path, "policy_version": None, + "checks": [], "runs": [], "ok": True} + try: + policy = ootb.load_policy(policy_path or None) + except Exception as e: # noqa: BLE001 + report["checks"].append( + Check("static", "policy file parses", False, + f"{type(e).__name__}: {e}").as_dict()) + report["ok"] = False + return report + report["policy_version"] = policy.get("policy_version") + + checks = validate_static(policy) + validate_decisions_synthetic(policy) + if examples: + checks += validate_decisions_examples(policy) + report["checks"] = [c.as_dict() for c in checks] + report["ok"] = all(c.ok for c in checks) + + if run: + records = validate_runs(policy_path) + report["runs"] = [r.as_dict() for r in records] + + return report + + +def format_report(report: dict) -> str: + lines = [] + lines.append("=" * 72) + lines.append(f"OOTB policy validation: {report['policy_file'] or '(default)'}") + lines.append(f"policy_version: {report['policy_version']}") + lines.append("=" * 72) + + by_layer = {} + for c in report["checks"]: + by_layer.setdefault(c["layer"], []).append(c) + for layer in ("static", "decision"): + layer_checks = by_layer.get(layer, []) + if not layer_checks: + continue + npass = sum(1 for c in layer_checks if c["ok"]) + lines.append(f"\n[{layer}] {npass}/{len(layer_checks)} passed") + for c in layer_checks: + mark = "PASS" if c["ok"] else "FAIL" + suffix = f" ({c['detail']})" if c["detail"] else "" + if not c["ok"] or c["detail"]: + lines.append(f" {mark} {c['name']}{suffix}") + + failures = [c for c in report["checks"] if not c["ok"]] + if failures: + lines.append(f"\n*** {len(failures)} CHECK FAILURE(S) ***") + for c in failures: + lines.append(f" - [{c['layer']}] {c['name']}: {c['detail']}") + + if report["runs"]: + lines.append("\n" + "-" * 72) + lines.append("Layer 3 runs (recorded; not auto-judged except the flags):") + for r in report["runs"]: + wt = f"{r['walltime']:.1f}s" if r["walltime"] is not None else "?" + extra = [] + if r["objective"] is not None: + extra.append(f"obj={r['objective']:.4g}") + if r["rel_gap"] is not None: + extra.append(f"gap={r['rel_gap']:.2%}") + if r["iterations"] is not None: + extra.append(f"iters={r['iterations']}") + lines.append(f" {r['example']:8} {r['mode']:10} rc={r['returncode']} " + f"{wt:>8} {' '.join(extra)}") + flagged = [r for r in report["runs"] if r["flagged"]] + lines.append("\n*** FLAGGED RUNS (review these) ***" if flagged + else "\nNo runs flagged.") + for r in flagged: + lines.append(f" !! {r['example']} [{r['mode']}]: {r['flag_reason']}") + + lines.append("\n" + ("OVERALL: PASS" if report["ok"] else "OVERALL: FAIL")) + return "\n".join(lines) + + +def main(argv=None): + p = argparse.ArgumentParser( + prog="python -m mpisppy.generic.ootb_validate", + description="Validate an OOTB policy file (see design doc sec. 8).") + p.add_argument("policy", nargs="?", default="", + help="policy file path (default: the shipped default policy)") + p.add_argument("--examples", action="store_true", + help="also run decision checks on probe-instantiated real " + "example models (needs a solver to instantiate; not CI)") + p.add_argument("--run", action="store_true", + help="also actually RUN the recommended configs (layer 3; " + "slow, needs a solver and mpiexec; never a CI gate)") + p.add_argument("--json", metavar="PATH", default=None, + help="write the machine-readable report to PATH") + args = p.parse_args(argv) + + report = run_validation(args.policy, examples=args.examples, run=args.run) + print(format_report(report)) + if args.json: + with open(args.json, "w") as fp: + json.dump(report, fp, indent=2) + print(f"\n[wrote JSON report to {args.json}]") + return 0 if report["ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mpisppy/generic/parsing.py b/mpisppy/generic/parsing.py index 951fc99e2..1f3a7213d 100644 --- a/mpisppy/generic/parsing.py +++ b/mpisppy/generic/parsing.py @@ -95,6 +95,21 @@ def load_module(model_fname): def parse_args(m): """Parse CLI args given the model module m. Returns a Config object.""" cfg = config.Config() + add_driver_args(cfg, m) + cfg.parse_command_line(f"mpi-sppy for {cfg.module_name}") + + cfg.checker() # looks for inconsistencies + return cfg + + +def add_driver_args(cfg, m=None): + """Declare every generic_cylinders option on cfg, WITHOUT parsing. + + Split out of parse_args so the same authoritative set of options can be + obtained without a command line -- the OOTB validator uses it (with m=None) + to learn which CLI flags are real. When m is given, its inparser_adder runs + too (the model-specific options). + """ cfg.proper_bundle_config() cfg.pickle_scenarios_config() cfg.pre_pickle_args() @@ -104,7 +119,8 @@ def parse_args(m): domain=str, default=None, argparse=True) - assert hasattr(m, "inparser_adder"), "The model file must have an inparser_adder function" + assert m is None or hasattr(m, "inparser_adder"), \ + "The model file must have an inparser_adder function" cfg.add_to_config(name="solution_base_name", description="The string used for a directory of ouput along with a csv and an npv file (default None, which means no soltion output)", domain=str, @@ -114,7 +130,8 @@ def parse_args(m): domain=str, default=None) - m.inparser_adder(cfg) + if m is not None: + m.inparser_adder(cfg) # many models, e.g., farmer, need num_scens_required # in which case, it should go in the inparser_adder function # cfg.num_scens_required() @@ -177,11 +194,6 @@ def parse_args(m): from mpisppy.generic.admm import admm_args admm_args(cfg) - cfg.parse_command_line(f"mpi-sppy for {cfg.module_name}") - - cfg.checker() # looks for inconsistencies - return cfg - def name_lists(module, cfg, bundle_wrapper=None): """Build all_scenario_names and all_nodenames from module and cfg. diff --git a/mpisppy/tests/test_ootb_validate.py b/mpisppy/tests/test_ootb_validate.py new file mode 100644 index 000000000..166ed9a34 --- /dev/null +++ b/mpisppy/tests/test_ootb_validate.py @@ -0,0 +1,116 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI gate for the out-of-the-box (OOTB) policy-file validator. + +Runs only the solver-free, fast layers -- layer 1 (static schema) and the +synthetic-facts subset of layer 2 (pure recommend() decisions on hand-built +Facts) -- on the shipped policy file(s). Example instantiation, anything needing +a solver, and all of layer 3 run nightly / on demand / locally, NOT here. See +doc/designs/out_of_the_box_design.md sec. 8. +""" + +import copy +import os +import unittest + +from mpisppy.generic import ootb_validate as val +from mpisppy.generic import out_of_the_box as ootb + + +def _shipped_policy_files(): + d = ootb._policies_dir() + return [os.path.join(d, f) for f in sorted(os.listdir(d)) + if f.startswith("ootb_policy_") and f.endswith(".json")] + + +class TestStaticAndSyntheticGate(unittest.TestCase): + """Every shipped policy must pass layers 1 + 2-synthetic cleanly.""" + + def test_every_shipped_policy_passes(self): + files = _shipped_policy_files() + self.assertTrue(files, "no shipped OOTB policy files found") + for path in files: + with self.subTest(policy=os.path.basename(path)): + report = val.run_validation(path) # layers 1 + 2-synthetic only + fails = [c for c in report["checks"] if not c["ok"]] + self.assertTrue( + report["ok"], + msg="validation failures:\n" + "\n".join( + f" [{c['layer']}] {c['name']}: {c['detail']}" for c in fails)) + + def test_default_policy_static_layer(self): + policy = ootb.load_policy() + fails = [c for c in val.validate_static(policy) if not c.ok] + self.assertEqual( + [], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + def test_default_policy_decision_layer(self): + policy = ootb.load_policy() + fails = [c for c in val.validate_decisions_synthetic(policy) if not c.ok] + self.assertEqual( + [], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + +class TestValidFlags(unittest.TestCase): + def test_known_flags_are_valid(self): + flags = val.valid_flags() + for f in ("--lagrangian", "--xhatshuffle", "--fwph", "--subgradient", + "--reduced-costs", "--xhatxbar", "--solver-name", + "--EF-solver-name", "--scenarios-per-bundle", "--grad-rho", + "--default-rho", "--rel-gap", "--max-iterations", + "--dynamic-rho-primal-crit", "--linearize-proximal-terms"): + self.assertIn(f, flags) + + def test_decomposition_flags_are_real(self): + flags = val.valid_flags() + bad = sorted(f for f in ootb.DECOMPOSITION_FLAGS if f not in flags) + self.assertEqual([], bad, msg=f"unknown DECOMPOSITION_FLAGS: {bad}") + + +class TestValidatorCatchesBadPolicies(unittest.TestCase): + """The validator must actually FAIL on malformed policies (not just pass).""" + + def setUp(self): + self.policy = copy.deepcopy(ootb.load_policy()) + + def _fails(self, checks): + return [c for c in checks if not c.ok] + + def test_missing_top_level_key(self): + del self.policy["solver"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_bogus_option_flag(self): + self.policy["option_categories"]["termination"]["flag"] = "--not-a-real-flag" + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_spoke_rung_not_a_real_flag(self): + self.policy["spoke_ladder"]["rungs"][0]["flag"] = "--bogus-spoke" + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_cold_start_guess_names_unknown_key(self): + # regression: a _cold_start_guess entry that is prose, not a real key. + self.policy["effort_scaling"]["_cold_start_guess"].append("not_a_key") + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_rho_setter_must_list_all_setters(self): + self.policy["option_categories"]["rho_setter"]["superseded_by"] = ["--grad-rho"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_decision_layer_detects_broken_ef_gate(self): + # If the rank floor is absurdly high, OOTB would never decompose; the + # "large problem -> decompose" decision check must then fail. + self.policy["ef_fallback"]["min_ranks_for_decomposition"] = 10**9 + self.assertTrue(self._fails(val.validate_decisions_synthetic(self.policy))) + + +if __name__ == "__main__": + unittest.main() diff --git a/run_coverage.bash b/run_coverage.bash index c89506f39..cff0d2b77 100755 --- a/run_coverage.bash +++ b/run_coverage.bash @@ -122,6 +122,9 @@ run_phase "test_smps (serial)" \ run_phase "test_generic_cylinders (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_generic_cylinders.py -v +run_phase "test_ootb_validate (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_validate.py -v + run_phase "test_jensens (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_jensens.py -v From 6a4b3d7580ba142bc0bbdffddcf366199a11570b Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 19:06:48 -0700 Subject: [PATCH 30/34] Out-of-the-box: effort-calibration tool (design sec. 9) Add mpisppy/generic/ootb_calibrate.py -- turns the abstract effort_scaling coefficients and EF budget into data-tuned, interpretable values from timed solves on the examples (the producer side of the dated-policy migration path). 1. Times extensive-form solves over a spread of bundle sizes across models of differing continuous / integer / nonant content (farmer / sizes / aircond, with calibration-tuned scenario counts so several bundle sizes are reachable). 2. Fits cont_coeff / int_weight / int_exponent / int_nonant_coeff by non-negative least squares per candidate integer exponent, choosing the exponent by R^2. 3. Calibrates effort to ~SECONDS: the fitted coefficients are kept in seconds units, so modeled effort approximates predicted solve time and the absolute EF budget reads as roughly seconds (ef_effort_budget set from ef_target_seconds) instead of an opaque large number. Significant-figure rounding preserves the legitimately tiny coefficients (int_weight ~ 1e-9 when the exponent is 3 and there are ~150 integers -- decimal rounding would zero it). 4. Writes a new dated policy with the fitted numbers and provenance in place of the cold-start guesses. CI test mpisppy/tests/test_ootb_calibrate.py covers only the PURE parts (solver-free): coefficient recovery from synthetic points, the minimum-points guard, significant-figure rounding, and that a fitted policy still passes the validator's static + synthetic-decision layers. Wired into run_coverage.bash and test_pr_and_main.yml. The measurement tier needs a solver and runs on demand / locally, like the validator's run tier. Verified on this box (gurobi): R^2 ~= 0.9995 over 14 timed solves; the resulting effort model keeps a continuous problem on the EF to ~370k scenarios but decomposes a 150-integer MIP after ~28 -- the MIP superlinearity is now real -- and the calibrated policy passes the validator. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/test_pr_and_main.yml | 4 + mpisppy/generic/ootb_calibrate.py | 327 +++++++++++++++++++++++++ mpisppy/tests/test_ootb_calibrate.py | 95 +++++++ run_coverage.bash | 3 + 4 files changed, 429 insertions(+) create mode 100644 mpisppy/generic/ootb_calibrate.py create mode 100644 mpisppy/tests/test_ootb_calibrate.py diff --git a/.github/workflows/test_pr_and_main.yml b/.github/workflows/test_pr_and_main.yml index 6e0221135..e7ad9963b 100644 --- a/.github/workflows/test_pr_and_main.yml +++ b/.github/workflows/test_pr_and_main.yml @@ -128,6 +128,10 @@ jobs: run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_validate.py -v + - name: Test out-of-the-box effort calibrator (pure fit) + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_calibrate.py -v + - name: Test xhat from file run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_xhat_from_file.py -v diff --git a/mpisppy/generic/ootb_calibrate.py b/mpisppy/generic/ootb_calibrate.py new file mode 100644 index 000000000..442ecf5ab --- /dev/null +++ b/mpisppy/generic/ootb_calibrate.py @@ -0,0 +1,327 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) effort-calibration tool. + +The effort_scaling coefficients and the EF effort budget are abstract by +construction (arbitrary "effort units"); left as hand-guesses they are +unassessable. This tool turns them into data-tuned, interpretable values from +timed solves on the mpi-sppy examples (the producer side of the dated-policy +migration path -- design doc sec. 9). It: + + 1. FITS the effort_scaling shape -- cont_coeff, int_weight, int_exponent, + int_nonant_coeff -- so that modeled effort tracks measured solve time, by + timing extensive-form solves over a spread of bundle sizes across models of + differing continuous / integer / nonant content. + 2. CALIBRATES effort to ~seconds: the fitted coefficients are kept in seconds + units, so modeled effort approximates predicted solve time and the absolute + budgets read as roughly seconds (ef_effort_budget is set from the policy's + ef_target_seconds) instead of opaque large numbers. + 3. WRITES a new dated policy file with the fitted numbers in place of the + cold-start guesses. + + python -m mpisppy.generic.ootb_calibrate --solver-name gurobi \\ + --output mpisppy/generic/ootb_policies/ootb_policy_2026-07-01.json + +Caveats: solve time is machine- and solver-dependent and (for MIPs) noisy, so a +calibrated scale is per reference machine/solver and approximate. The fit is the +LINEAR part (cont/int/nonant coefficients) for each candidate integer exponent; +the best exponent is chosen by R^2. The pure fit (fit_effort_model) is +solver-free and unit-tested; the measurement needs a solver and is not run in CI. +""" + +from __future__ import annotations + +import argparse +import datetime +import json +import sys +import time + +import mpisppy.utils.sputils as sputils +from mpisppy.generic import out_of_the_box as ootb +from mpisppy.generic import ootb_validate as val + + +# Integer-exponent grid searched during the fit (int_exponent in the effort +# model). 1.0 = integers scale like LP; > 1 captures branch-and-bound blow-up. +EXPONENT_GRID = [1.0, 1.25, 1.5, 1.75, 2.0, 2.5, 3.0] + +# Bundle sizes (scenarios per EF solve) used to probe how solve time scales. +DEFAULT_SPB_GRID = [1, 2, 4, 8, 16] + +# How many times to solve each point (the minimum is kept -- a denoiser). +DEFAULT_REPS = 2 + + +# --------------------------------------------------------------------------- +# Fitting [pure; unit-tested] +# --------------------------------------------------------------------------- + + +def _round_sig(x: float, n: int = 6) -> float: + """Round to n significant figures. Decimal-place rounding would destroy the + legitimately tiny coefficients that multiply huge (int*spb)^exponent terms + (e.g. int_weight ~ 1e-9 when the exponent is 3 and there are ~150 integers).""" + import math + if x == 0 or not math.isfinite(x): + return 0.0 + return round(x, -int(math.floor(math.log10(abs(x)))) + (n - 1)) + + +def _design_columns(point: dict, exponent: float): + """The three effort features for one measured point, matching ootb._effort: + continuous content (~spb), integer content (^exponent), integer nonants + (fixed per bundle).""" + cont = point["vars_cont"] * point["spb"] + nint = point["vars_int"] * point["spb"] + return [cont, nint ** exponent, point["nonants_int"]] + + +def fit_effort_model(points: list, exponents=EXPONENT_GRID) -> dict: + """Fit the effort_scaling coefficients + an effort->seconds scale to timed + solves. + + points: list of dicts with PER-SCENARIO `vars_cont`, `vars_int`, + `nonants_int`, the bundle size `spb`, and the measured `seconds`. + + For each candidate integer exponent we solve a non-negative least squares for + (a, b, c) so that a*cont + b*int^p + c*nonant ~= seconds, and keep the + exponent with the best R^2. The fitted coefficients are kept in SECONDS units + (we do NOT divide the scale back out), so modeled effort approximates the + predicted solve time directly -- the absolute budgets (ef_effort_budget) then + read as roughly seconds (e.g. ~120) instead of opaque large numbers. + seconds_per_effort_unit is therefore ~1; it stays in the schema to document + the units and to let a focus rescale. Returns the effort_scaling fields plus + r2 / n_points. + """ + import numpy as np + from scipy.optimize import nnls + + if len(points) < 3: + raise ValueError(f"need at least 3 measured points to fit, got {len(points)}") + + y = np.array([p["seconds"] for p in points], dtype=float) + best = None + for p in exponents: + A = np.array([_design_columns(pt, p) for pt in points], dtype=float) + coef, _ = nnls(A, y) + ss_res = float(((A @ coef - y) ** 2).sum()) + ss_tot = float(((y - y.mean()) ** 2).sum()) + r2 = 1.0 - ss_res / ss_tot if ss_tot > 0 else 0.0 + if best is None or r2 > best["r2"]: + best = {"coef": coef, "p": p, "r2": r2} + + a, b, c = (float(x) for x in best["coef"]) + return { + "cont_coeff": _round_sig(a), + "int_weight": _round_sig(b), + "int_exponent": best["p"], + "int_nonant_coeff": _round_sig(c), + "seconds_per_effort_unit": 1.0, # effort is calibrated to ~seconds + "r2": round(best["r2"], 4), + "n_points": len(points), + } + + +# --------------------------------------------------------------------------- +# Measurement [needs a solver; not run in CI] +# --------------------------------------------------------------------------- + + +def _pick_solver(policy: dict, requested: str | None) -> str: + """A non-persistent solver for one-shot EF solves (persistent interfaces + need set_instance and add setup overhead that would pollute the timing).""" + if requested: + return requested + for name in policy["solver"]["preference_order"]: + plain = name.replace("_persistent", "") + if plain in ootb._detect_available_solvers([plain]): + return plain + raise RuntimeError("no known solver available; pass --solver-name") + + +def _time_ef_solve(names, scenario_creator, kwargs, solver_name, reps) -> float: + """Build an EF over `names` scenarios and return the MIN wall-clock solve + time over `reps` solves (model build excluded from the timing).""" + import pyomo.environ as pyo + best = None + for _ in range(reps): + ef = sputils.create_EF(names, scenario_creator, kwargs, + suppress_warnings=True) + solver = pyo.SolverFactory(solver_name) + t0 = time.perf_counter() + solver.solve(ef) + dt = time.perf_counter() - t0 + best = dt if best is None else min(best, dt) + return best + + +def measure_example(spec: dict, solver_name: str, spb_grid, reps) -> list: + """Probe one example's per-scenario size profile, then time EF solves over a + range of bundle sizes. Returns a list of measured points.""" + module = val._load_example_module(spec) + cfg = val._example_cfg(spec, module) + kwargs = module.kw_creator(cfg) + profile = ootb._size_profile(ootb._build_probe_scenario(module, cfg)) + total = ootb._detect_num_scens(module, cfg) + points = [] + for spb in spb_grid: + if spb > total: + break + names = module.scenario_names_creator(spb) + try: + seconds = _time_ef_solve(names, module.scenario_creator, kwargs, + solver_name, reps) + except Exception as e: # noqa: BLE001 + print(f" [{spec['name']} spb={spb}] solve failed: " + f"{type(e).__name__}: {e}", file=sys.stderr) + continue + pt = {"example": spec["name"], "spb": spb, + "vars_cont": profile["vars_cont"], "vars_int": profile["vars_int"], + "nonants_int": profile["nonants_int"], "seconds": round(seconds, 6)} + points.append(pt) + print(f" {spec['name']:8} spb={spb:<3} " + f"cont={profile['vars_cont']} int={profile['vars_int']} " + f"nonants_int={profile['nonants_int']} -> {seconds:.4f}s") + return points + + +def _calibration_specs() -> list: + """Example specs with scenario counts tuned for calibration: enough + scenarios that several bundle sizes are reachable (the validator's run-tier + deliberately uses tiny counts; calibration wants a wider spread, and more + integer points come from a larger MIP EF).""" + counts = {"farmer": {"num_scens": 16}, "sizes": {"num_scens": 10}, + "aircond": {"branching_factors": [4, 4]}} + specs = [] + for spec in val.example_models(): + s = dict(spec) + s["scens"] = counts.get(spec["name"], spec["scens"]) + specs.append(s) + return specs + + +def collect_points(policy: dict, solver_name: str, spb_grid, reps, + specs=None) -> list: + points = [] + for spec in (specs if specs is not None else _calibration_specs()): + print(f"[calibrate] timing {spec['name']} ({spec['kind']}) ...") + points.extend(measure_example(spec, solver_name, spb_grid, reps)) + return points + + +# --------------------------------------------------------------------------- +# Apply the fit to a policy +# --------------------------------------------------------------------------- + + +def calibrated_policy(base_policy: dict, fit: dict, points: list, + solver_name: str, today: str) -> dict: + """Return a copy of base_policy with the fitted effort_scaling, the + effort->seconds scale, and a seconds-derived ef_effort_budget.""" + import copy + pol = copy.deepcopy(base_policy) + + es = pol["effort_scaling"] + es["cont_coeff"] = fit["cont_coeff"] + es["int_weight"] = fit["int_weight"] + es["int_exponent"] = fit["int_exponent"] + es["int_nonant_coeff"] = fit["int_nonant_coeff"] + es["seconds_per_effort_unit"] = round(fit["seconds_per_effort_unit"], 8) + es["_calibration"] = { + "solver": solver_name, "r2": fit["r2"], "n_points": fit["n_points"], + "date": today, "note": "Fitted by mpisppy.generic.ootb_calibrate on the " + "example set; per reference machine/solver and approximate. " + "seconds_per_effort_unit: modeled effort * this ~= seconds."} + es.pop("_cold_start_guess", None) # these numbers are now data-tuned + + # Derive the EF budget in effort units from the seconds target so the budget + # is consistent with the fitted scale (effort <= seconds / sec_per_effort). + ef = pol["ef_fallback"] + spe = fit["seconds_per_effort_unit"] + if spe > 0 and ef.get("ef_target_seconds"): + ef["ef_effort_budget"] = int(round(ef["ef_target_seconds"] / spe)) + ef["_calibration_note"] = ("ef_effort_budget derived from " + "ef_target_seconds / seconds_per_effort_unit " + f"(calibrated {today}).") + guesses = ef.get("_cold_start_guess", []) + ef["_cold_start_guess"] = [g for g in guesses if g != "ef_effort_budget"] + + pol["policy_version"] = today + pol["provenance"] = (f"CALIBRATED {today} by mpisppy.generic.ootb_calibrate " + f"(solver {solver_name}, R^2={fit['r2']}, " + f"{fit['n_points']} timed EF solves on the example set). " + "effort_scaling and ef_effort_budget are data-tuned; " + "remaining numbers are still authored. Per reference " + "machine/solver and approximate (MIP times are noisy).") + return pol + + +# --------------------------------------------------------------------------- +# Orchestration + CLI +# --------------------------------------------------------------------------- + + +def run_calibration(base_policy_path, solver_name=None, spb_grid=DEFAULT_SPB_GRID, + reps=DEFAULT_REPS, today=None): + """Measure, fit, and return (calibrated_policy_dict, fit, points).""" + base = ootb.load_policy(base_policy_path or None) + solver = _pick_solver(base, solver_name) + print(f"[calibrate] solver: {solver}") + points = collect_points(base, solver, spb_grid, reps) + fit = fit_effort_model(points) + if today is None: + today = datetime.date.today().isoformat() + pol = calibrated_policy(base, fit, points, solver, today) + return pol, fit, points + + +def main(argv=None): + p = argparse.ArgumentParser( + prog="python -m mpisppy.generic.ootb_calibrate", + description="Calibrate OOTB effort_scaling from timed example solves " + "(design doc sec. 9).") + p.add_argument("--base", default="", + help="base policy to start from (default: shipped default)") + p.add_argument("--solver-name", default=None, + help="solver for the timed solves (default: first available)") + p.add_argument("--output", default=None, + help="write the calibrated policy here (default: print only)") + p.add_argument("--reps", type=int, default=DEFAULT_REPS, + help=f"solves per point, min kept (default {DEFAULT_REPS})") + p.add_argument("--spb", default=None, + help="comma-separated bundle sizes (default " + f"{','.join(map(str, DEFAULT_SPB_GRID))})") + args = p.parse_args(argv) + + spb_grid = ([int(x) for x in args.spb.split(",")] if args.spb + else DEFAULT_SPB_GRID) + pol, fit, points = run_calibration(args.base, args.solver_name, spb_grid, + args.reps) + + print("\n[calibrate] fit:") + for k in ("cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff", + "seconds_per_effort_unit", "r2", "n_points"): + print(f" {k}: {fit[k]}") + print(f" ef_effort_budget -> {pol['ef_fallback']['ef_effort_budget']} " + f"(from {pol['ef_fallback'].get('ef_target_seconds')}s target)") + + if args.output: + with open(args.output, "w") as fp: + json.dump(pol, fp, indent=2) + print(f"\n[calibrate] wrote calibrated policy to {args.output}") + print("[calibrate] validate it with: python -m mpisppy.generic." + f"ootb_validate {args.output}") + else: + print("\n[calibrate] (no --output given; not writing a policy file)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mpisppy/tests/test_ootb_calibrate.py b/mpisppy/tests/test_ootb_calibrate.py new file mode 100644 index 000000000..e77c7f8fd --- /dev/null +++ b/mpisppy/tests/test_ootb_calibrate.py @@ -0,0 +1,95 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI test for the OOTB effort-calibration tool's PURE parts. + +Only the fitting / policy-assembly logic is exercised here -- it is solver-free. +The measurement (timed example solves) needs a solver and is run on demand / +locally, not in CI. See doc/designs/out_of_the_box_design.md sec. 9. +""" + +import unittest + +from mpisppy.generic import ootb_calibrate as cal +from mpisppy.generic import ootb_validate as val +from mpisppy.generic import out_of_the_box as ootb + + +def _synthetic_points(cont_coeff, int_weight, exponent, int_nonant_coeff): + """Points generated from a known effort model (zero measurement noise).""" + pts = [] + for (vc, vi, ni) in [(10, 0, 0), (5, 3, 2), (0, 8, 5), (20, 1, 1)]: + for spb in (1, 2, 4, 8): + cont = vc * spb + nint = vi * spb + seconds = (cont_coeff * cont + int_weight * (nint ** exponent) + + int_nonant_coeff * ni) + pts.append({"vars_cont": vc, "vars_int": vi, "nonants_int": ni, + "spb": spb, "seconds": seconds}) + return pts + + +class TestFit(unittest.TestCase): + def test_recovers_known_coefficients(self): + truth = dict(cont_coeff=0.01, int_weight=0.5, exponent=2.0, + int_nonant_coeff=0.2) + fit = cal.fit_effort_model(_synthetic_points(**truth)) + self.assertAlmostEqual(fit["cont_coeff"], 0.01, places=5) + self.assertAlmostEqual(fit["int_weight"], 0.5, places=5) + self.assertEqual(fit["int_exponent"], 2.0) + self.assertAlmostEqual(fit["int_nonant_coeff"], 0.2, places=5) + self.assertGreater(fit["r2"], 0.999) + self.assertEqual(fit["seconds_per_effort_unit"], 1.0) + + def test_requires_minimum_points(self): + with self.assertRaises(ValueError): + cal.fit_effort_model([{"vars_cont": 1, "vars_int": 0, + "nonants_int": 0, "spb": 1, "seconds": 1.0}]) + + def test_round_sig_preserves_tiny_values(self): + # decimal rounding would zero this; significant-figure rounding keeps it. + self.assertAlmostEqual(cal._round_sig(1.5618e-09), 1.5618e-09) + self.assertEqual(cal._round_sig(0.0), 0.0) + self.assertAlmostEqual(cal._round_sig(123456.789, 3), 123000.0) + + +class TestCalibratedPolicy(unittest.TestCase): + def setUp(self): + self.base = ootb.load_policy() + self.fit = cal.fit_effort_model(_synthetic_points( + cont_coeff=0.01, int_weight=0.5, exponent=2.0, int_nonant_coeff=0.2)) + + def test_effort_scaling_replaced(self): + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + es = pol["effort_scaling"] + self.assertAlmostEqual(es["cont_coeff"], 0.01, places=5) + self.assertEqual(es["int_exponent"], 2.0) + self.assertIn("seconds_per_effort_unit", es) + self.assertNotIn("_cold_start_guess", es) # numbers are now data-tuned + self.assertEqual(pol["policy_version"], "2026-07-01") + + def test_ef_budget_is_seconds_like(self): + # budget = target seconds / seconds_per_effort_unit (~1), so it reads as + # seconds rather than an opaque huge number. + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + self.assertEqual(pol["ef_fallback"]["ef_effort_budget"], + pol["ef_fallback"]["ef_target_seconds"]) + + def test_calibrated_policy_passes_static_validation(self): + # end-to-end: a fitted policy must still be well-formed. + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + fails = [c for c in val.validate_static(pol) if not c.ok] + self.assertEqual([], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + fails = [c for c in val.validate_decisions_synthetic(pol) if not c.ok] + self.assertEqual([], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + +if __name__ == "__main__": + unittest.main() diff --git a/run_coverage.bash b/run_coverage.bash index cff0d2b77..3dc8889d9 100755 --- a/run_coverage.bash +++ b/run_coverage.bash @@ -125,6 +125,9 @@ run_phase "test_generic_cylinders (serial)" \ run_phase "test_ootb_validate (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_validate.py -v +run_phase "test_ootb_calibrate (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_calibrate.py -v + run_phase "test_jensens (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_jensens.py -v From 0a16078fad3991691a90841d753e3c8d8361521e Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 19:11:25 -0700 Subject: [PATCH 31/34] Out-of-the-box: ship calibrated numbers in the policy Regenerate the shipped policy with ootb_calibrate (gurobi, R^2~=0.9995 over 14 timed EF solves on the example set), so the v1 numbers are the calibrator's output rather than hand-guesses (design sec. 9): - effort_scaling coefficients are data-tuned and in ~seconds units; - ef_effort_budget is 120 (derived from ef_target_seconds), reading as roughly seconds instead of the old opaque 100000; - provenance + an effort_scaling._calibration block record the solver, R^2, point count, and the per-machine/solver caveat. These numbers are this reference machine's; the calibration is reproducible by re-running the tool, and a later dated file folds in a broader corpus. Also make the calibrator reconcile the prose it supersedes: the effort_scaling and ef_fallback _comment fields no longer claim "All cold-start guesses" for the now-calibrated coefficients / budget. Fix test_ootb_validate's cold_start_guess regression test to inject into bundle_sizing (which stays hand-authored) instead of effort_scaling (whose _cold_start_guess the calibrator now removes). Co-Authored-By: Claude Opus 4.8 (1M context) --- mpisppy/generic/ootb_calibrate.py | 9 + .../ootb_policies/ootb_policy_2026-06-28.json | 201 +++++++++++++----- mpisppy/tests/test_ootb_validate.py | 3 +- 3 files changed, 159 insertions(+), 54 deletions(-) diff --git a/mpisppy/generic/ootb_calibrate.py b/mpisppy/generic/ootb_calibrate.py index 442ecf5ab..f4272e3da 100644 --- a/mpisppy/generic/ootb_calibrate.py +++ b/mpisppy/generic/ootb_calibrate.py @@ -240,6 +240,11 @@ def calibrated_policy(base_policy: dict, fit: dict, points: list, "example set; per reference machine/solver and approximate. " "seconds_per_effort_unit: modeled effort * this ~= seconds."} es.pop("_cold_start_guess", None) # these numbers are now data-tuned + # Reconcile the prose: the coefficients are no longer cold-start guesses. + es["_comment"] = es.get("_comment", "").replace( + "All cold-start guesses; foci ship different shapes; benchmark data refines.", + "Coefficients are calibrated by ootb_calibrate (see _calibration); foci " + "may ship different shapes; more benchmark data refines further.") # Derive the EF budget in effort units from the seconds target so the budget # is consistent with the fitted scale (effort <= seconds / sec_per_effort). @@ -252,6 +257,10 @@ def calibrated_policy(base_policy: dict, fit: dict, points: list, f"(calibrated {today}).") guesses = ef.get("_cold_start_guess", []) ef["_cold_start_guess"] = [g for g in guesses if g != "ef_effort_budget"] + ef["_comment"] = ef.get("_comment", "").replace( + "All cold-start guesses.", + "ef_effort_budget is calibrated (see _calibration_note); " + "ef_target_seconds and ef_if_num_scens_at_most remain authored guesses.") pol["policy_version"] = today pol["provenance"] = (f"CALIBRATED {today} by mpisppy.generic.ootb_calibrate " diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index b7b035cbf..4c203f58f 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -1,60 +1,115 @@ { "_description": "Out-of-the-box (OOTB) auto-configuration policy. This is DATA, not code: a thin Python interpreter reads it and a set of facts about the environment (MPI ranks, installed solvers, optional cores/memory) and the model (scenario count, stage structure) to assemble a defensible mpi-sppy run. User-supplied options always win; OOTB only fills gaps. See doc/designs/out_of_the_box_design.md.", - "schema_version": 1, "policy_version": "2026-06-28", - "provenance": "COLD START. Authored by hand from mpi-sppy defaults and common practice; NO benchmark data yet. Every numeric threshold below is an educated guess flagged with _cold_start_guess and is expected to be replaced by data-tuned values in a later dated policy file.", - + "provenance": "CALIBRATED 2026-06-28 by mpisppy.generic.ootb_calibrate (solver gurobi, R^2=0.9995, 14 timed EF solves on the example set). effort_scaling and ef_effort_budget are data-tuned; remaining numbers are still authored. Per reference machine/solver and approximate (MIP times are noisy).", "ef_fallback": { - "_comment": "When to solve the monolithic EF instead of decomposing. Always EF when ranks < min_ranks_for_decomposition (requirement 3). Otherwise EF when the WHOLE problem is small enough, measured with the SAME effort model as bundle sizing (effort_scaling) applied to all scenarios as one model, i.e. effort(num_scens). Unlike bundle sizing's RELATIVE budget, the EF budget is ABSOLUTE (the monolith has no single-scenario reference). base: effort(num_scens) <= ef_effort_budget (same effort units as bundle effort). plus: measured_single_scenario_seconds * effort(num_scens)/effort(1) <= ef_target_seconds. minus (no size profile): fall back to the count rule num_scens <= ef_if_num_scens_at_most. All cold-start guesses.", + "_comment": "When to solve the monolithic EF instead of decomposing. Always EF when ranks < min_ranks_for_decomposition (requirement 3). Otherwise EF when the WHOLE problem is small enough, measured with the SAME effort model as bundle sizing (effort_scaling) applied to all scenarios as one model, i.e. effort(num_scens). Unlike bundle sizing's RELATIVE budget, the EF budget is ABSOLUTE (the monolith has no single-scenario reference). base: effort(num_scens) <= ef_effort_budget (same effort units as bundle effort). plus: measured_single_scenario_seconds * effort(num_scens)/effort(1) <= ef_target_seconds. minus (no size profile): fall back to the count rule num_scens <= ef_if_num_scens_at_most. ef_effort_budget is calibrated (see _calibration_note); ef_target_seconds and ef_if_num_scens_at_most remain authored guesses.", "min_ranks_for_decomposition": 3, "ef_if_num_scens_at_most": 2, - "ef_effort_budget": 100000, + "ef_effort_budget": 120, "ef_target_seconds": 120, - "_cold_start_guess": ["ef_if_num_scens_at_most", "ef_effort_budget", "ef_target_seconds"] + "_cold_start_guess": [ + "ef_if_num_scens_at_most", + "ef_target_seconds" + ], + "_calibration_note": "ef_effort_budget derived from ef_target_seconds / seconds_per_effort_unit (calibrated 2026-06-28)." }, - "solver": { "_comment": "Picked by trying preference_order in order and keeping the first that SolverFactory(name).available(exception_flag=False) reports. Persistence is requested by NAMING the *_persistent interface (no separate flag). Persistent commercial first, then plain commercial, then free QP-capable, then LP/MIP-only (which forces prox linearization).", "preference_order": [ - "gurobi_persistent", "cplex_persistent", "xpress_persistent", - "gurobi", "cplex", "xpress", - "appsi_highs", "highs", "ipopt", - "cbc", "glpk" + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent", + "gurobi", + "cplex", + "xpress", + "appsi_highs", + "highs", + "ipopt", + "cbc", + "glpk" ], "commercial": [ - "gurobi", "cplex", "xpress", - "gurobi_persistent", "cplex_persistent", "xpress_persistent" + "gurobi", + "cplex", + "xpress", + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent" ], "qp_capable": [ - "gurobi", "cplex", "xpress", - "gurobi_persistent", "cplex_persistent", "xpress_persistent", - "ipopt", "highs", "appsi_highs" + "gurobi", + "cplex", + "xpress", + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent", + "ipopt", + "highs", + "appsi_highs" + ], + "lp_mip_only_force_linearize_prox": [ + "glpk", + "cbc" ], - "lp_mip_only_force_linearize_prox": ["glpk", "cbc"], "caveats": { - "_comment": "Surfaced via suggestions, not branched on yet (model integrality is not introspected in v1; see design §6).", + "_comment": "Surfaced via suggestions, not branched on yet (model integrality is not introspected in v1; see design \u00a76).", "ipopt": "Continuous NLP only; cannot handle integer variables.", "highs": "QP ok; mixed-integer quadratic (MIQP) support is limited, so the PH prox on MIP subproblems may be rejected.", "glpk_cbc": "LP/MIP only; cannot handle the quadratic PH prox term, so OOTB sets --linearize-proximal-terms." } }, - "hub": { "_comment": "PH is the default hub in the generic driver; no flag is emitted to select it.", "default_factory": "ph_hub", "flag": null }, - "spoke_ladder": { "_comment": "Ordered list of spokes to switch on as ranks allow, lowest priority number first. Only flags WIRED in mpisppy/generic/spokes.py appear here. The 3-rank minimum fills priorities 1 and 2 (one outer + one inner bound) so there is always an optimality gap. Each rung consumes one cylinder; see rank_allocation for how leftover ranks are spent. NOTE: the ORDERING of priorities 3-6 is itself a cold-start guess (no benchmark data yet); _cold_start_guess lists only real keys, so it cannot name the ordering, but treat the priority order below as provisional.", "rungs": [ - {"priority": 1, "flag": "--lagrangian", "factory": "lagrangian_spoke", "bound": "outer", "note": "Lagrangian dual bound; cheap and robust. Core."}, - {"priority": 2, "flag": "--xhatshuffle", "factory": "xhatshuffle_spoke", "bound": "inner", "note": "Incumbent finder over shuffled scenarios. Core."}, - {"priority": 3, "flag": "--fwph", "factory": "fwph_spoke", "bound": "outer", "note": "Frank-Wolfe PH; often tightens the outer bound."}, - {"priority": 4, "flag": "--xhatxbar", "factory": "xhatxbar_spoke", "bound": "inner", "note": "Cheap second incumbent source from xbar."}, - {"priority": 5, "flag": "--subgradient", "factory": "subgradient_spoke", "bound": "outer", "note": "Alternative dual bound."}, - {"priority": 6, "flag": "--reduced-costs","factory": "reduced_costs_spoke","bound": "outer", "note": "More advanced; benefits markedly from a persistent solver. Last."} + { + "priority": 1, + "flag": "--lagrangian", + "factory": "lagrangian_spoke", + "bound": "outer", + "note": "Lagrangian dual bound; cheap and robust. Core." + }, + { + "priority": 2, + "flag": "--xhatshuffle", + "factory": "xhatshuffle_spoke", + "bound": "inner", + "note": "Incumbent finder over shuffled scenarios. Core." + }, + { + "priority": 3, + "flag": "--fwph", + "factory": "fwph_spoke", + "bound": "outer", + "note": "Frank-Wolfe PH; often tightens the outer bound." + }, + { + "priority": 4, + "flag": "--xhatxbar", + "factory": "xhatxbar_spoke", + "bound": "inner", + "note": "Cheap second incumbent source from xbar." + }, + { + "priority": 5, + "flag": "--subgradient", + "factory": "subgradient_spoke", + "bound": "outer", + "note": "Alternative dual bound." + }, + { + "priority": 6, + "flag": "--reduced-costs", + "factory": "reduced_costs_spoke", + "bound": "outer", + "note": "More advanced; benefits markedly from a persistent solver. Last." + } ], "core_roster_min": { "_comment": "Minimum spokes for a meaningful run: at least one of each bound type. With the PH hub this is the 3-rank floor.", @@ -62,9 +117,10 @@ "inner": 1 }, "max_cylinders": 7, - "_cold_start_guess": ["max_cylinders"] + "_cold_start_guess": [ + "max_cylinders" + ] }, - "rank_allocation": { "_comment": "Two parts. (A) ROSTER SIZE: prefer a SMALL CORE (>=1 outer + >=1 inner spoke) widened by ranks over a wide roster of weak spokes. Start from the core; add further ladder rungs only while each cylinder would still get >= min_ranks_per_cylinder ranks (uniform/nominal basis -- a coarse gate) and we stay <= max_cylinders. So 6 ranks -> hub + lagrangian + xhatshuffle (3 cylinders, widened), NOT 6 single-rank cylinders. (B) UNBALANCED DISTRIBUTION (flex-ranks): split ranks across the chosen cylinders by per-cylinder RATIOS, not uniformly -- cheaper cylinders get fewer ranks. ratios are normalized over the chosen cylinders and floored at 1 rank each. WARNING: rank_ratios here are a CRUDE cold-start (e.g. xhatter 0.2); the right split is a much more complicated calculation that depends on the NATURE OF THE SUBPROBLEMS (relative solve cost), ideally informed by the plus tier's measurements.", "min_ranks_per_cylinder": 2, @@ -74,18 +130,27 @@ "--xhatxbar": 0.2, "--xhatlshaped": 0.2 }, - "_cold_start_guess": ["min_ranks_per_cylinder", "default_rank_ratio", "rank_ratios"] + "_cold_start_guess": [ + "min_ranks_per_cylinder", + "default_rank_ratio", + "rank_ratios" + ] }, - "effort_scaling": { - "_comment": "SHAPE of how solve effort grows with the size of a (sub)problem. Shared by bundle sizing at every tier (design doc Bundle sizing section). effort(n) for a bundle of n scenarios = cont_coeff*(n*vars_cont) + int_weight*(n*vars_int)^int_exponent + int_nonant_coeff*nonants_int, using the probe size profile (vars_cont, vars_int, nonants_int). The base tier uses this RELATIVE (ratio vs a single scenario); the plus tier uses the SAME shape CALIBRATED by a measured single-scenario solve time. int_exponent>1 captures MIP superlinearity; =1 treats integers like LP. All cold-start guesses; foci ship different shapes; benchmark data refines.", - "cont_coeff": 1.0, - "int_weight": 10.0, - "int_exponent": 2.0, - "int_nonant_coeff": 5.0, - "_cold_start_guess": ["cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff"] + "_comment": "SHAPE of how solve effort grows with the size of a (sub)problem. Shared by bundle sizing at every tier (design doc Bundle sizing section). effort(n) for a bundle of n scenarios = cont_coeff*(n*vars_cont) + int_weight*(n*vars_int)^int_exponent + int_nonant_coeff*nonants_int, using the probe size profile (vars_cont, vars_int, nonants_int). The base tier uses this RELATIVE (ratio vs a single scenario); the plus tier uses the SAME shape CALIBRATED by a measured single-scenario solve time. int_exponent>1 captures MIP superlinearity; =1 treats integers like LP. Coefficients are calibrated by ootb_calibrate (see _calibration); foci may ship different shapes; more benchmark data refines further.", + "cont_coeff": 2.46332e-05, + "int_weight": 1.57234e-09, + "int_exponent": 3.0, + "int_nonant_coeff": 5.96467e-05, + "seconds_per_effort_unit": 1.0, + "_calibration": { + "solver": "gurobi", + "r2": 0.9995, + "n_points": 14, + "date": "2026-06-28", + "note": "Fitted by mpisppy.generic.ootb_calibrate on the example set; per reference machine/solver and approximate. seconds_per_effort_unit: modeled effort * this ~= seconds." + } }, - "bundle_sizing": { "_comment": "How big should proper bundles be (not just whether to bundle)? Proper bundling only (loose bundling removed in 2026); --scenarios-per-bundle MUST divide num_scens; library requires number_of_bundles >= number_of_ranks. The MINUS tier CANNOT bundle: with no size profile there is no safe way to size a bundle, so minus always runs unbundled (bundling needs base or plus). For base/plus: pick the LARGEST scenarios_per_bundle that (a) divides num_scens, (b) leaves >= max(intra_ranks, min_bundles_per_intra_rank*intra_ranks) bundles, and (c) keeps a bundle's modeled effort within budget. base budget is RELATIVE: effort(spb)/effort(1) <= base_max_hardness_vs_single_scenario (unitless, portable, no measurement). plus budget is ABSOLUTE: measured_single_scenario_seconds * effort(spb)/effort(1) <= plus_target_seconds_per_bundle, with the measuring solve capped at plus_probe_solve_time_cap_seconds. Skip bundling below min_scens_to_consider_bundling. MIP solve times are noisy/non-monotone, so the shape in effort_scaling acts as a prior; a single measurement only sets the scale.", "min_scens_to_consider_bundling": 50, @@ -94,44 +159,74 @@ "base_max_hardness_vs_single_scenario": 10, "plus_target_seconds_per_bundle": 30, "plus_probe_solve_time_cap_seconds": 60, - "_cold_start_guess": ["min_scens_to_consider_bundling", "min_bundles_per_intra_rank", "base_max_hardness_vs_single_scenario", "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"] + "_cold_start_guess": [ + "min_scens_to_consider_bundling", + "min_bundles_per_intra_rank", + "base_max_hardness_vs_single_scenario", + "plus_target_seconds_per_bundle", + "plus_probe_solve_time_cap_seconds" + ] }, - "option_categories": { "_comment": "Named option blocks OOTB applies by default, one per CONCERN, beyond the structural choices (solver / EF / spokes / bundling). Each is applied UNLESS the user already set any flag in its 'superseded_by' list -- the robust, per-concern form of 'user options win' (the user may address a concern with a DIFFERENT flag than OOTB's default). 'value' null = a boolean flag. These are decomposition-run options (skipped on the EF path). Foci may change them. All cold-start.", "base_rho": { - "flag": "--default-rho", "value": "1", - "superseded_by": ["--default-rho"], + "flag": "--default-rho", + "value": "1", + "superseded_by": [ + "--default-rho" + ], "_note": "PH needs a base rho value. The grad-rho setter REFINES rho but does not itself supply a base, and (unlike sep/coeff/sensi rho) mpi-sppy does not auto-default it, so a decomposition with --grad-rho and no --default-rho is a hard error. Set a sane base here; the rho setter adapts from it." }, "rho_setter": { - "flag": "--grad-rho", "value": null, - "superseded_by": ["--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho", "--reduced-costs-rho"], + "flag": "--grad-rho", + "value": null, + "superseded_by": [ + "--grad-rho", + "--sensi-rho", + "--coeff-rho", + "--sep-rho", + "--reduced-costs-rho" + ], "_note": "mpi-sppy enforces that only ONE rho setter is active, so OOTB adding --grad-rho when the user already picked another rho setter would be a HARD ERROR -- this is exactly why superseded_by lists all of them, not just --grad-rho." }, "termination": { - "flag": "--rel-gap", "value": "0.01", - "superseded_by": ["--rel-gap", "--abs-gap"] + "flag": "--rel-gap", + "value": "0.01", + "superseded_by": [ + "--rel-gap", + "--abs-gap" + ] }, "max_iterations": { - "flag": "--max-iterations", "value": "100", - "superseded_by": ["--max-iterations"] + "flag": "--max-iterations", + "value": "100", + "superseded_by": [ + "--max-iterations" + ] }, "dynamic_rho": { - "flag": "--dynamic-rho-primal-crit", "value": null, - "superseded_by": ["--dynamic-rho-primal-crit", "--dynamic-rho-dual-crit"], + "flag": "--dynamic-rho-primal-crit", + "value": null, + "superseded_by": [ + "--dynamic-rho-primal-crit", + "--dynamic-rho-dual-crit" + ], "_note": "Boolean flag (no argument needed); its paired threshold dynamic_rho_primal_thresh defaults to 0.1. Requires an active rho setter (config errors otherwise), which OOTB's default --grad-rho provides." }, - "_cold_start_guess": ["base_rho", "rho_setter", "termination", "max_iterations", "dynamic_rho"] + "_cold_start_guess": [ + "base_rho", + "rho_setter", + "termination", + "max_iterations", + "dynamic_rho" + ] }, - "additional_options": { "_comment": "Catch-all for any OTHER extra flags not worth a named category. Each entry: flag, value (null = boolean), and optional superseded_by (defaults to [its own flag], i.e. plain per-flag override). For concern-level override (e.g. rho setters) use option_categories instead. Decomposition-run options (skipped on the EF path).", "options": [] }, - "suggestions": { "_comment": "Requirement 4: a prioritized 'Suggestions' list, printed AFTER the run (so it can also reflect how the run went). The messages are MOSTLY COMPUTED in code (suggestion generators in mpisppy/generic/out_of_the_box.py): each inspects facts/decision/run-outcome and builds text with live values, so the PROSE lives in code, not here. Note the split from design sec. 5: decisions stay data-driven (this policy); suggestions are diagnostics and are computed. This block only TOGGLES/TUNES them -- 'disabled' lists generator names to suppress; thresholds (if any) can be added here for generators to read.", "disabled": [] } -} +} \ No newline at end of file diff --git a/mpisppy/tests/test_ootb_validate.py b/mpisppy/tests/test_ootb_validate.py index 166ed9a34..388401beb 100644 --- a/mpisppy/tests/test_ootb_validate.py +++ b/mpisppy/tests/test_ootb_validate.py @@ -98,7 +98,8 @@ def test_spoke_rung_not_a_real_flag(self): def test_cold_start_guess_names_unknown_key(self): # regression: a _cold_start_guess entry that is prose, not a real key. - self.policy["effort_scaling"]["_cold_start_guess"].append("not_a_key") + # (bundle_sizing stays hand-authored even after effort calibration.) + self.policy["bundle_sizing"]["_cold_start_guess"].append("not_a_key") self.assertTrue(self._fails(val.validate_static(self.policy))) def test_rho_setter_must_list_all_setters(self): From 763dd2c4e1986c666ea2a30870cf0fe30c64cb0b Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sun, 28 Jun 2026 19:16:17 -0700 Subject: [PATCH 32/34] Out-of-the-box: quick-start docs and a dedicated page Document OOTB as the recommended on-ramp (requirement 1). - New doc/src/out_of_the_box.rst: spirit, basic usage, what OOTB decides (solver, EF-vs-decompose, widened spoke core, flex-rank split, bundling, extra defaults), transparency (equivalent command line + Suggestions), the effort tiers and the seconds-calibrated EF gate, --inspect-only (incl. the assumed rank count for HPC planning), policy files and selection, and the validator and calibrator tools. - quick_start.rst: add "Recommended first run: let mpi-sppy configure itself" at the top of the farmer section, pointing to the full page. - generic_cylinders.rst: a tip box pointing newcomers at --out-of-the-box. - index.rst: add the page to the "Running with generic_cylinders" toctree. The examples are honest about the calibrated policy: on a fast machine with a commercial solver the small bundled examples solve as the EF within budget, so plain --out-of-the-box correctly chooses the EF for them; the note shows how to exercise the cylinder path (request a spoke, or use --out-of-the-box-minus). --out-of-the-box-plus is documented as reserved (currently behaves like base). Docs build clean (no new warnings). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/src/generic_cylinders.rst | 6 + doc/src/index.rst | 1 + doc/src/out_of_the_box.rst | 254 ++++++++++++++++++++++++++++++++++ doc/src/quick_start.rst | 18 +++ 4 files changed, 279 insertions(+) create mode 100644 doc/src/out_of_the_box.rst diff --git a/doc/src/generic_cylinders.rst b/doc/src/generic_cylinders.rst index ba189d077..ed6515ba9 100644 --- a/doc/src/generic_cylinders.rst +++ b/doc/src/generic_cylinders.rst @@ -8,6 +8,12 @@ run mpi-sppy. It provides command-line access to the hub-and-spoke system, the extensive form solver, confidence intervals, and many other features without requiring you to write a driver program. +.. tip:: + If you are new to mpi-sppy, add ``--out-of-the-box`` and let the driver + pick a sensible configuration automatically, then read the equivalent + command line it prints to learn the explicit options. See + :ref:`out_of_the_box`. + Your Model File (Module) ------------------------ diff --git a/doc/src/index.rst b/doc/src/index.rst index 71c846cb3..26a6e2843 100644 --- a/doc/src/index.rst +++ b/doc/src/index.rst @@ -25,6 +25,7 @@ MPI is used. :caption: Running with generic_cylinders generic_cylinders.rst + out_of_the_box.rst examples.rst ef.rst chance_constraints.rst diff --git a/doc/src/out_of_the_box.rst b/doc/src/out_of_the_box.rst new file mode 100644 index 000000000..1589b269c --- /dev/null +++ b/doc/src/out_of_the_box.rst @@ -0,0 +1,254 @@ +.. _out_of_the_box: + +Out-of-the-box auto-configuration +================================= + +The ``--out-of-the-box`` option lets a relatively new user obtain a *sensible* +mpi-sppy run with almost no knowledge of the library's internals. You supply a +model module (and its scenario data); mpi-sppy introspects the environment and +the model and assembles a defensible configuration -- algorithm, solver, spokes, +flexible rank split, and proper bundling -- instead of requiring a hand-crafted +hub/spoke command line. + +The spirit is *"here is my model, go,"* followed by a clear explanation of what +was chosen and how to do better. + +.. note:: + Out-of-the-box (OOTB) only *fills gaps*. **Any option you set explicitly + always wins** -- OOTB never overrides it. So you can start from + ``--out-of-the-box`` and override individual choices as you learn. + +Basic usage +----------- + +Add ``--out-of-the-box`` to an otherwise minimal ``generic_cylinders`` command +line (your model still needs its scenario count -- ``--num-scens`` for two-stage +problems, ``--branching-factors`` for multistage): + +.. code-block:: bash + + # serial -> too few ranks for cylinders, so OOTB solves the EF + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box + + # 3+ ranks available; OOTB decomposes when the problem is big/hard enough + mpiexec -np 3 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 6 --out-of-the-box + +OOTB prints the configuration it chose, the **equivalent explicit command line** +(so you can reproduce, learn from, and tweak it), runs the model, and then prints +a prioritized **Suggestions** list. + +.. note:: + OOTB decomposes only when it expects the decomposition to pay off. The base + tier estimates how long the monolithic EF would take and, if that is within + budget, solves the EF even when several ranks are available -- because for a + small or fast-solving model the EF *is* the right call. On a fast machine with + a commercial solver, the bundled examples (farmer, sizes, aircond) are cheap + enough that ``--out-of-the-box`` chooses the EF for all of them. To exercise + the cylinder path on a small problem, either request a spoke (any + decomposition flag, e.g. ``--lagrangian``, forces a decomposition) or use the + ``--out-of-the-box-minus`` tier, which has no size estimate and decomposes + whenever there are more than a couple of scenarios: + + .. code-block:: bash + + mpiexec -np 3 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 6 --out-of-the-box-minus + +What OOTB decides +----------------- + +In order, OOTB chooses: + +#. **Solver.** The first installed solver in a preference order (persistent + commercial, then commercial, then a free QP-capable solver, then LP/MIP-only). + An LP/MIP-only solver (cbc, glpk) automatically adds + ``--linearize-proximal-terms`` because it cannot take the quadratic PH prox. + If you pass ``--solver-name`` it is used as-is (and carried over to + ``--EF-solver-name`` if OOTB ends up solving the EF). +#. **Extensive form vs. decomposition.** With fewer than three ranks there is no + useful cylinder configuration (hub + at least two spokes), so OOTB solves the + **EF**. Above the rank floor, OOTB still solves the EF when the whole problem + is small enough to expect a quick monolithic solve (see *Effort and the EF + gate* below). Otherwise it decomposes. If you explicitly request a + decomposition (any spoke or a non-default hub) and have enough ranks, OOTB + never substitutes the EF. +#. **Spokes (a small, widened core).** Starting from a minimal core of one outer + bound (``--lagrangian``) and one inner/incumbent spoke (``--xhatshuffle``), + OOTB adds further spokes only while every cylinder would still keep at least a + couple of ranks. The preference is to give a few cylinders width rather than + pile on many weak single-rank spokes. So six ranks become three cylinders, + widened -- not six single-rank cylinders. +#. **Flexible rank split.** Ranks are split across cylinders unevenly: cheaper + cylinders (the xhat family) get a smaller share via per-spoke + ``--*-rank-ratio`` flags. (This is a crude cold-start split; the right split + depends on relative subproblem solve cost.) +#. **Proper bundling.** When there are many scenarios, OOTB forms proper bundles, + choosing the largest ``--scenarios-per-bundle`` that divides the scenario + count, leaves at least as many bundles as ranks, and keeps a bundle's modeled + solve effort within budget. +#. **A few extra defaults**, each backed off if you addressed the same concern: + ``--default-rho 1`` and the ``--grad-rho`` rho setter, ``--rel-gap 0.01``, + ``--max-iterations 100``, and ``--dynamic-rho-primal-crit``. + +Transparency +------------ + +OOTB prints the choices and the equivalent command line up front, and a +**Suggestions** list after the run (so the suggestions can reflect how the run +went). For example, a serial farmer run reports:: + + [out-of-the-box] tier 'base', policy 2026-06-28 + - solver: gurobi_persistent (first available in preference order) + - --EF: only 1 ranks; decomposition needs >= 3 + - --EF-solver-name gurobi_persistent: EF solver (gurobi_persistent) + [out-of-the-box] equivalent command line: + mpiexec -np 1 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi_persistent + ... + [out-of-the-box] Suggestions: + * Ran the monolithic EF because only 1 MPI rank(s) were available; with + >= 3 ranks OOTB would decompose (hub + bound spokes). + +The equivalent command line is anchored with the module and scenario +specification and lists every flag OOTB added, so you can paste it (dropping +``--out-of-the-box``) to reproduce or modify the run. + +Effort tiers (how deeply OOTB inspects the model) +------------------------------------------------- + +Three mutually-exclusive flags select how deeply OOTB looks at the model; they +share one interpreter and one policy file and differ only in how much they +inspect. Every decision uses the best fact available and degrades gracefully to +a suggestion when a fact is missing. + +.. list-table:: + :header-rows: 1 + :widths: 22 14 64 + + * - Flag + - Instantiates + - What it can decide + * - ``--out-of-the-box-minus`` + - nothing + - EF gate by scenario *count*; solver by availability. Cannot size proper + bundles (no model size information). + * - ``--out-of-the-box`` *(default)* + - one probe scenario + - Size-aware EF gate; integrality- and size-aware bundling. The recommended + tier. + * - ``--out-of-the-box-plus`` + - *(reserved)* + - Planned: instantiate all scenarios and do a brief timed solve for + solve-time / gap information. Currently behaves like the base tier. + +``--out-of-the-box-plus`` is **not** an autotuner: it is "out-of-the-box with +more information," making the same one-shot decisions as the base tier. (It is +reserved for a future release; today it is equivalent to ``--out-of-the-box``.) + +Effort and the EF gate +^^^^^^^^^^^^^^^^^^^^^^^ + +The base tier instantiates one scenario to read its size profile (continuous / +integer variable counts and nonant counts) and models solve *effort* from it. +The numbers in the policy file are **calibrated to roughly seconds**, so the EF +budget reads as a wall-clock target: OOTB solves the EF when the whole problem's +modeled effort is within ``ef_effort_budget`` (about that many seconds), and +sizes bundles so a bundle is at most a small multiple as hard as a single +scenario. Integer content scales superlinearly, so an integer-heavy model +decomposes at a far smaller scenario count than a continuous one. + +``--inspect-only`` (dry run) +---------------------------- + +``--inspect-only`` does the inspection, prints the configuration, the equivalent +command line, and config-time suggestions, then **stops before the production +run**. It is independent of OOTB (on its own it just verifies that one scenario +instantiates), but pairs naturally with it: + +.. code-block:: bash + + # plan the run, print the equivalent command line, do not solve + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box --inspect-only + +``--inspect-only`` takes an optional **assumed rank count** for HPC planning: +``--inspect-only 512`` plans as if 512 ranks were available -- so you can get the +recommended command line for a large job *from a login node, without launching +it*. Everything else (installed solvers, model size) still comes from the real +session; only the rank count is hypothetical. + +Policy files +------------ + +OOTB's choices are driven by a dated, declarative **policy file** under +``mpisppy/generic/ootb_policies/`` -- data, not code, interpreted by a thin +Python routine so every decision is explainable. A bare ``--out-of-the-box`` +uses the newest shipped default policy; ``--out-of-the-box PATH`` uses the policy +file at ``PATH`` (the optional value of the flag *is* the path). Policy files +with different *foci* may ship side by side, distinguished by filename; you +select one by passing its path. The run logs which policy and ``policy_version`` +it used. + +The policy holds the solver preference order, the EF budget, the spoke ladder and +rank ratios, the bundle-effort model, and the extra-option defaults. Its numbers +are produced by the calibration tool (below), not hand-guessed. + +.. _ootb_validator: + +Validating a policy file +------------------------ + +A policy file is checked by a validator that confirms it is well-formed and that +its recommendations make sense -- and, on demand, actually run: + +.. code-block:: bash + + # static schema + decision checks on the default policy + python -m mpisppy.generic.ootb_validate + + # also exercise the real example models (needs a solver to instantiate) + python -m mpisppy.generic.ootb_validate --examples + + # also actually run the recommended configs and flag problem cases + python -m mpisppy.generic.ootb_validate --run --json report.json + +The validator has three layers: static schema checks (every referenced flag is +real, keys/types are right); decision checks (the EF gate fires when it should, +forced decomposition wins, bundling is valid, no conflicting rho setters); and +run checks that execute the recommended configurations and **flag** two cases for +a human to review -- an EF that misses a 1% gap in ten minutes, and cylinders +that max out on iterations. It produces a human-readable and a machine-readable +(``--json``) report. The fast, solver-free layers gate continuous integration; +the run tier is for nightly / local use. + +.. _ootb_calibrator: + +Calibrating the effort numbers +------------------------------ + +The effort coefficients and budget are produced by a calibration tool from timed +solves on the example models, so they track measured wall-clock time on a +reference machine rather than being guesses: + +.. code-block:: bash + + python -m mpisppy.generic.ootb_calibrate --solver-name gurobi \ + --output mpisppy/generic/ootb_policies/ootb_policy_.json + +The tool times extensive-form solves over a spread of bundle sizes, fits the +continuous / integer / nonant coefficients (choosing the integer exponent by best +fit), keeps the coefficients in seconds units so the budgets read as seconds, and +writes a new dated policy with provenance. Solve time is machine- and +solver-dependent (and noisy for MIPs), so a calibrated policy is per reference +machine and approximate; re-run the tool to recalibrate for your environment. + +Limitations +----------- + +OOTB aims for a *defensible* configuration, not an *optimal* one. It does not +tune convergence parameters to optimality and does not search a parameter space. +For full control, write an explicit hub/spoke command line (the rest of +:ref:`generic_cylinders` documents every option) -- and remember that OOTB emits +exactly such a command line for you to start from. diff --git a/doc/src/quick_start.rst b/doc/src/quick_start.rst index 76b6eea09..b2b31e666 100644 --- a/doc/src/quick_start.rst +++ b/doc/src/quick_start.rst @@ -377,6 +377,24 @@ guidance and HPC-specific tips, see :ref:`Install mpi4py`. Running the Farmer Example --------------------------- +**Recommended first run: let mpi-sppy configure itself.** Add +``--out-of-the-box`` and the driver introspects the environment and the model and +picks a sensible configuration (solver, EF vs. decomposition, spokes, bundling), +prints the equivalent explicit command line, and runs it: + +.. code-block:: bash + + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box + +Any option you set explicitly always wins, so this is a good starting point you +can refine. (For a small, fast-solving model like farmer, OOTB will sensibly +choose the extensive form; it decomposes for larger or harder problems.) See +:ref:`out_of_the_box` for the full description -- effort tiers, +``--inspect-only``, policy files, and the validation and calibration tools. + +The explicit forms below show what such a run is equivalent to. + **Solve the EF** (does not use MPI): .. code-block:: bash From edba4273c899ca077f7d3f86e7cc076304581ecf Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Mon, 29 Jun 2026 09:00:38 -0700 Subject: [PATCH 33/34] Out-of-the-box: raise CI coverage of the new modules (codecov) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OOTB modules were ~54% patch coverage because large parts only run with a solver / MPI / a subprocess, which CI does not have. Close the gap two ways, without pretending to test what cannot run in CI: - New mpisppy/tests/test_out_of_the_box.py drives the interpreter wiring (gather_facts probe -> recommend -> apply_decision -> configure) on farmer with --inspect-only N for a deterministic rank count. Building a scenario does not solve, so the whole EF / decomposition / bundling decision-and-apply path runs solver-free, plus direct tests of apply_decision, command_line, the suggestion generators, requested/effort_and_policy, and _user_flags. - Extend the validator/calibrator tests to cover the remaining pure logic: format_report, _parse_decompose (both branches), main (pass/fail), _scen_cli, _child_env, example_models, validate_decisions_examples (scenario build, no solve), and _calibration_specs / _design_columns. - # pragma: no cover on the genuinely un-CI-able code: the validator run tier (validate_runs, _run_one — subprocess + solver), the calibrator measurement (_pick_solver, _time_ef_solve, measure_example, collect_points, run_calibration), the __main__ guards, and the generic_cylinders OOTB CLI hook (configure() itself is unit-tested). Coverage of the new modules: ootb_calibrate 100%, ootb_validate 95%, out_of_the_box 94%. test_out_of_the_box wired into run_coverage.bash and test_pr_and_main.yml. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/test_pr_and_main.yml | 4 + mpisppy/generic/ootb_calibrate.py | 14 +- mpisppy/generic/ootb_validate.py | 6 +- mpisppy/generic_cylinders.py | 6 +- mpisppy/tests/test_ootb_calibrate.py | 12 ++ mpisppy/tests/test_ootb_validate.py | 79 +++++++++ mpisppy/tests/test_out_of_the_box.py | 228 +++++++++++++++++++++++++ run_coverage.bash | 3 + 8 files changed, 339 insertions(+), 13 deletions(-) create mode 100644 mpisppy/tests/test_out_of_the_box.py diff --git a/.github/workflows/test_pr_and_main.yml b/.github/workflows/test_pr_and_main.yml index e7ad9963b..1d797377d 100644 --- a/.github/workflows/test_pr_and_main.yml +++ b/.github/workflows/test_pr_and_main.yml @@ -124,6 +124,10 @@ jobs: run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_generic_cylinders.py -v + - name: Test out-of-the-box interpreter wiring + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_out_of_the_box.py -v + - name: Test out-of-the-box policy validator (layers 1 + 2-synthetic) run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_validate.py -v diff --git a/mpisppy/generic/ootb_calibrate.py b/mpisppy/generic/ootb_calibrate.py index f4272e3da..2085fc06b 100644 --- a/mpisppy/generic/ootb_calibrate.py +++ b/mpisppy/generic/ootb_calibrate.py @@ -134,7 +134,7 @@ def fit_effort_model(points: list, exponents=EXPONENT_GRID) -> dict: # --------------------------------------------------------------------------- -def _pick_solver(policy: dict, requested: str | None) -> str: +def _pick_solver(policy: dict, requested: str | None) -> str: # pragma: no cover """A non-persistent solver for one-shot EF solves (persistent interfaces need set_instance and add setup overhead that would pollute the timing).""" if requested: @@ -146,7 +146,7 @@ def _pick_solver(policy: dict, requested: str | None) -> str: raise RuntimeError("no known solver available; pass --solver-name") -def _time_ef_solve(names, scenario_creator, kwargs, solver_name, reps) -> float: +def _time_ef_solve(names, scenario_creator, kwargs, solver_name, reps) -> float: # pragma: no cover """Build an EF over `names` scenarios and return the MIN wall-clock solve time over `reps` solves (model build excluded from the timing).""" import pyomo.environ as pyo @@ -162,7 +162,7 @@ def _time_ef_solve(names, scenario_creator, kwargs, solver_name, reps) -> float: return best -def measure_example(spec: dict, solver_name: str, spb_grid, reps) -> list: +def measure_example(spec: dict, solver_name: str, spb_grid, reps) -> list: # pragma: no cover """Probe one example's per-scenario size profile, then time EF solves over a range of bundle sizes. Returns a list of measured points.""" module = val._load_example_module(spec) @@ -208,7 +208,7 @@ def _calibration_specs() -> list: def collect_points(policy: dict, solver_name: str, spb_grid, reps, - specs=None) -> list: + specs=None) -> list: # pragma: no cover points = [] for spec in (specs if specs is not None else _calibration_specs()): print(f"[calibrate] timing {spec['name']} ({spec['kind']}) ...") @@ -278,7 +278,7 @@ def calibrated_policy(base_policy: dict, fit: dict, points: list, def run_calibration(base_policy_path, solver_name=None, spb_grid=DEFAULT_SPB_GRID, - reps=DEFAULT_REPS, today=None): + reps=DEFAULT_REPS, today=None): # pragma: no cover """Measure, fit, and return (calibrated_policy_dict, fit, points).""" base = ootb.load_policy(base_policy_path or None) solver = _pick_solver(base, solver_name) @@ -291,7 +291,7 @@ def run_calibration(base_policy_path, solver_name=None, spb_grid=DEFAULT_SPB_GRI return pol, fit, points -def main(argv=None): +def main(argv=None): # pragma: no cover p = argparse.ArgumentParser( prog="python -m mpisppy.generic.ootb_calibrate", description="Calibrate OOTB effort_scaling from timed example solves " @@ -332,5 +332,5 @@ def main(argv=None): return 0 -if __name__ == "__main__": +if __name__ == "__main__": # pragma: no cover sys.exit(main()) diff --git a/mpisppy/generic/ootb_validate.py b/mpisppy/generic/ootb_validate.py index 51b59de75..e5ab0c524 100644 --- a/mpisppy/generic/ootb_validate.py +++ b/mpisppy/generic/ootb_validate.py @@ -505,7 +505,7 @@ def _parse_decompose(out: str): return converged, iters, rel_gap -def _run_one(spec, mode, nranks, extra_args, timeout): +def _run_one(spec, mode, nranks, extra_args, timeout): # pragma: no cover """Run one example via generic_cylinders in a subprocess and time it.""" if mode == "EF": cmd = [sys.executable, "-m", "mpisppy.generic_cylinders"] @@ -533,7 +533,7 @@ def _run_one(spec, mode, nranks, extra_args, timeout): def validate_runs(policy_path: str, *, ef_time_limit=EF_TIME_LIMIT_SEC, - ef_gap=EF_GAP_TARGET) -> list: + ef_gap=EF_GAP_TARGET) -> list: # pragma: no cover """Run two configurations per example and flag the two failure modes.""" records = [] pol_arg = ["--out-of-the-box", policy_path] if policy_path else ["--out-of-the-box"] @@ -684,5 +684,5 @@ def main(argv=None): return 0 if report["ok"] else 1 -if __name__ == "__main__": +if __name__ == "__main__": # pragma: no cover sys.exit(main()) diff --git a/mpisppy/generic_cylinders.py b/mpisppy/generic_cylinders.py index 9ee1a34d4..3beacd48f 100644 --- a/mpisppy/generic_cylinders.py +++ b/mpisppy/generic_cylinders.py @@ -51,12 +51,12 @@ # driver path below runs it. --inspect-only stops before the production run. from mpisppy.generic import out_of_the_box as ootb ootb_state = None - if ootb.requested(cfg): + if ootb.requested(cfg): # pragma: no cover (CLI entrypoint; configure() is unit-tested) ootb_state = ootb.configure(module, cfg) if cfg.get("inspect_only") is not None: ootb.report_suggestions(ootb_state) # config-time suggestions only sys.exit(0) - elif cfg.get("inspect_only") is not None: + elif cfg.get("inspect_only") is not None: # pragma: no cover (CLI entrypoint) ootb.inspect_only_standalone(module, cfg) sys.exit(0) @@ -163,5 +163,5 @@ def scenario_denouement(rank, sname, s): # Out-of-the-box: the prioritized "Suggestions" list is printed AFTER the # run so it can also reflect how the run went (req. 4). - if ootb_state is not None: + if ootb_state is not None: # pragma: no cover (CLI entrypoint) ootb.report_suggestions(ootb_state) diff --git a/mpisppy/tests/test_ootb_calibrate.py b/mpisppy/tests/test_ootb_calibrate.py index e77c7f8fd..e9bd94cea 100644 --- a/mpisppy/tests/test_ootb_calibrate.py +++ b/mpisppy/tests/test_ootb_calibrate.py @@ -91,5 +91,17 @@ def test_calibrated_policy_passes_static_validation(self): msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) +class TestCalibrationSpecs(unittest.TestCase): + def test_specs_use_larger_counts(self): + specs = cal._calibration_specs() + self.assertEqual({s["name"] for s in specs}, + {"farmer", "sizes", "aircond"}) + farmer = next(s for s in specs if s["name"] == "farmer") + self.assertEqual(farmer["scens"], {"num_scens": 16}) + self.assertEqual(cal._design_columns( + {"vars_cont": 2, "vars_int": 3, "nonants_int": 4, "spb": 5}, 2.0), + [10, (15) ** 2.0, 4]) + + if __name__ == "__main__": unittest.main() diff --git a/mpisppy/tests/test_ootb_validate.py b/mpisppy/tests/test_ootb_validate.py index 388401beb..f883aeabd 100644 --- a/mpisppy/tests/test_ootb_validate.py +++ b/mpisppy/tests/test_ootb_validate.py @@ -113,5 +113,84 @@ def test_decision_layer_detects_broken_ef_gate(self): self.assertTrue(self._fails(val.validate_decisions_synthetic(self.policy))) +class TestExamplesLayer(unittest.TestCase): + """The decision-on-real-models layer builds scenarios (no solve), so it runs + here; the farmer checks must pass.""" + + def test_validate_decisions_examples_farmer_passes(self): + checks = val.validate_decisions_examples(ootb.load_policy()) + self.assertTrue(checks) + farmer = [c for c in checks if "farmer" in c.name] + self.assertTrue(farmer) + self.assertTrue(all(c.ok for c in farmer), + msg="\n".join(f"{c.name}: {c.detail}" + for c in farmer if not c.ok)) + + +class TestHelpers(unittest.TestCase): + def test_scen_cli(self): + self.assertEqual(val._scen_cli({"scens": {"num_scens": 6}}), + ["--num-scens", "6"]) + self.assertEqual( + val._scen_cli({"scens": {"branching_factors": [3, 2]}}), + ["--branching-factors", "3 2"]) # one space-joined token + + def test_child_env_scrubs_mpi_vars(self): + os.environ["OMPI_TESTVAR"] = "1" + try: + self.assertNotIn("OMPI_TESTVAR", val._child_env()) + self.assertIn("PATH", val._child_env()) + finally: + del os.environ["OMPI_TESTVAR"] + + def test_example_models_paths_exist(self): + for spec in val.example_models(): + self.assertTrue(os.path.isdir(spec["dir"]), spec["dir"]) + + def test_parse_decompose_converged(self): + out = ("[ 0.8] Terminating based on inter-cylinder relative gap 0.9%\n" + "[ 0.8] Statistics at termination\n" + "[ 0.8] Iter. Best Bound Best Incumbent Rel. Gap Abs. Gap\n" + "[ 0.8] 99 -124.0 -123.0 0.910% 1.1\n") + converged, iters, gap = val._parse_decompose(out) + self.assertTrue(converged) + self.assertEqual(iters, 99) + self.assertAlmostEqual(gap, 0.0091, places=4) + + def test_parse_decompose_maxed_out(self): + out = ("[ 9.0] Statistics at termination\n" + "[ 9.0] 100 -10.0 -5.0 50.000% 5.0\n") + converged, iters, gap = val._parse_decompose(out) + self.assertFalse(converged) + self.assertEqual(iters, 100) + + +class TestReportAndMain(unittest.TestCase): + def test_format_report_covers_branches(self): + report = { + "policy_file": "p.json", "policy_version": "2026-06-28", "ok": False, + "checks": [{"layer": "static", "name": "a", "ok": True, "detail": ""}, + {"layer": "decision", "name": "b", "ok": False, + "detail": "boom"}], + "runs": [{"example": "farmer", "mode": "EF", "returncode": 0, + "walltime": 0.5, "objective": -1.0, "rel_gap": None, + "iterations": None, "flagged": False, "flag_reason": ""}, + {"example": "sizes", "mode": "decompose", "returncode": 0, + "walltime": 1.0, "objective": None, "rel_gap": 0.5, + "iterations": 100, "flagged": True, + "flag_reason": "maxed out"}], + } + text = val.format_report(report) + self.assertIn("CHECK FAILURE", text) + self.assertIn("FLAGGED RUNS", text) + self.assertIn("OVERALL: FAIL", text) + + def test_main_passes_on_default_policy(self): + self.assertEqual(val.main([]), 0) # layers 1+2 on default + + def test_main_fails_on_missing_policy(self): + self.assertEqual(val.main(["/no/such/policy.json"]), 1) + + if __name__ == "__main__": unittest.main() diff --git a/mpisppy/tests/test_out_of_the_box.py b/mpisppy/tests/test_out_of_the_box.py new file mode 100644 index 000000000..56a194b27 --- /dev/null +++ b/mpisppy/tests/test_out_of_the_box.py @@ -0,0 +1,228 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI tests for the out-of-the-box (OOTB) interpreter wiring. + +Solver-free: the OOTB decision/apply path and the base-tier probe only need to +*build* a scenario (no solve), so the whole gather_facts -> recommend -> +apply_decision -> configure flow runs in CI on the farmer example. The +environment rank count is supplied via --inspect-only N so the EF and +decomposition branches are both exercised deterministically without launching +mpiexec. (The solver-dependent run/measurement tiers are exercised on demand / +locally; see test_ootb_validate / test_ootb_calibrate.) +""" + +import os +import sys +import unittest + +import mpisppy.utils.config as config +from mpisppy.generic import parsing +from mpisppy.generic import out_of_the_box as ootb + + +def _repo_root(): + here = os.path.dirname(os.path.abspath(__file__)) + return os.path.dirname(os.path.dirname(here)) + + +_FARMER_DIR = os.path.join(_repo_root(), "examples", "farmer") + + +def _farmer_module(): + if _FARMER_DIR not in sys.path: + sys.path.insert(0, _FARMER_DIR) + import farmer + return farmer + + +def _farmer_cfg(num_scens=6, **overrides): + module = _farmer_module() + cfg = config.Config() + parsing.add_driver_args(cfg, module) + cfg.num_scens = num_scens + cfg.module_name = "farmer" + for k, v in overrides.items(): + cfg[k] = v + return cfg, module + + +class TestConfigureNoSolver(unittest.TestCase): + """Drive configure() (probe + recommend + apply) without solving.""" + + def setUp(self): + self._argv = sys.argv + sys.argv = ["prog", "--module-name", "farmer", "--num-scens", "6"] + + def tearDown(self): + sys.argv = self._argv + + def test_base_few_ranks_picks_ef(self): + cfg, module = _farmer_cfg(out_of_the_box="", inspect_only="1") + state = ootb.configure(module, cfg) + self.assertTrue(state.decision.run_ef) + self.assertEqual(state.decision.ef_reason, "min_ranks") + self.assertTrue(cfg.EF) # apply_decision set it + # the probe populated a size profile (farmer is continuous) + self.assertIsNotNone(state.facts.vars_cont) + self.assertEqual(state.facts.vars_int, 0) + + def test_minus_tier_decomposes_by_count(self): + cfg, module = _farmer_cfg(out_of_the_box_minus="", inspect_only="6") + state = ootb.configure(module, cfg) + self.assertFalse(state.decision.run_ef) + self.assertTrue(cfg.lagrangian and cfg.xhatshuffle) + self.assertIsNone(state.facts.vars_cont) # minus: no probe + # minus cannot bundle + self.assertIsNone(cfg.get("scenarios_per_bundle")) + + def test_base_forced_decomposition_bundles(self): + # user --lagrangian forces decomposition even though the problem is small + sys.argv = sys.argv + ["--lagrangian"] + cfg, module = _farmer_cfg(num_scens=60, out_of_the_box="", + inspect_only="3", lagrangian=True) + state = ootb.configure(module, cfg) + self.assertFalse(state.decision.run_ef) + spb = cfg.get("scenarios_per_bundle") + self.assertIsNotNone(spb) # 60 scens -> bundles + self.assertEqual(60 % int(spb), 0) + + def test_report_suggestions_runs(self): + cfg, module = _farmer_cfg(out_of_the_box="", inspect_only="1") + state = ootb.configure(module, cfg) + ootb.report_suggestions(state) # config-time suggestions + self.assertIsInstance(state.decision.suggestions, list) + + +class TestInspectStandalone(unittest.TestCase): + def test_verify_instantiation_and_standalone(self): + cfg, module = _farmer_cfg() + profile = ootb.verify_instantiation(module, cfg) + self.assertEqual(set(profile), + {"vars_int", "vars_cont", "nonants_total", "nonants_int"}) + self.assertGreater(profile["vars_cont"], 0) + ootb.inspect_only_standalone(module, cfg) # prints, returns None + + +class TestApplyDecision(unittest.TestCase): + def test_apply_sets_cfg_values(self): + cfg, _ = _farmer_cfg() + d = ootb.Decision() + d.args = [ootb.ChosenArg("--lagrangian", None, "x"), + ootb.ChosenArg("--solver-name", "gurobi", "x"), + ootb.ChosenArg("--scenarios-per-bundle", "10", "x"), + ootb.ChosenArg("--rel-gap", "0.01", "x")] + ootb.apply_decision(d, cfg) + self.assertTrue(cfg.lagrangian) + self.assertEqual(cfg.solver_name, "gurobi") + self.assertEqual(cfg.scenarios_per_bundle, 10) + self.assertAlmostEqual(cfg.rel_gap, 0.01) + + def test_ef_solver_name_carried_over(self): + cfg, _ = _farmer_cfg() + d = ootb.Decision(run_ef=True) + d.args = [ootb.ChosenArg("--solver-name", "cplex", "x")] + ootb.apply_decision(d, cfg) + self.assertTrue(cfg.EF) + self.assertEqual(cfg.EF_solver_name, "cplex") # belt-and-suspenders + + +class TestCommandLineAndFlags(unittest.TestCase): + def test_command_line_two_stage(self): + facts = ootb.Facts("farmer", 3, set(), 6) + d = ootb.Decision() + d.args = [ootb.ChosenArg("--lagrangian", None, "x"), + ootb.ChosenArg("--solver-name", "gurobi", "x")] + cl = d.command_line(facts) + self.assertIn("--module-name farmer", cl) + self.assertIn("--num-scens 6", cl) + self.assertIn("--lagrangian", cl) + self.assertIn("-np 3", cl) + + def test_command_line_multistage(self): + facts = ootb.Facts("aircond", 3, set(), 6, multistage=True, + branching_factors=[3, 2]) + cl = ootb.Decision().command_line(facts) + self.assertIn("--branching-factors 3 2", cl) + self.assertNotIn("--num-scens", cl) + + def test_requested_and_effort_and_policy(self): + cfg, _ = _farmer_cfg(out_of_the_box="") + self.assertTrue(ootb.requested(cfg)) + effort, path = ootb.effort_and_policy(cfg) + self.assertEqual(effort, "base") + self.assertIsNone(path) + cfg2, _ = _farmer_cfg() + self.assertFalse(ootb.requested(cfg2)) + self.assertEqual(ootb.effort_and_policy(cfg2), (None, None)) + + def test_two_tiers_is_an_error(self): + cfg, _ = _farmer_cfg(out_of_the_box="", out_of_the_box_minus="") + with self.assertRaises(RuntimeError): + ootb.effort_and_policy(cfg) + + def test_user_flags_from_argv(self): + saved = sys.argv + try: + sys.argv = ["p", "--lagrangian", "--max-iterations=50", "-q"] + flags = ootb._user_flags() + self.assertIn("--lagrangian", flags) + self.assertIn("--max-iterations", flags) # =value stripped + self.assertNotIn("-q", flags) # single dash ignored + finally: + sys.argv = saved + + +class TestSuggestionGenerators(unittest.TestCase): + """Exercise each computed suggestion generator.""" + + def setUp(self): + self.policy = ootb.load_policy() + + def _msgs(self, d, facts, outcome=None): + return ootb.make_suggestions(d, facts, self.policy, outcome) + + def test_few_ranks_and_minus(self): + facts = ootb.Facts("m", 1, set(), 3, effort="minus") + d = ootb.Decision(run_ef=True, ef_reason="min_ranks") + self.assertTrue(any("only 1 MPI" in m for m in self._msgs(d, facts))) + + def test_no_persistent_and_more_ranks(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + msgs = self._msgs(d, facts) + self.assertTrue(any("persistent" in m for m in msgs)) + self.assertTrue(any("cylinders configured" in m for m in msgs)) + + def test_linearized_prox(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="cbc", num_cylinders=3) + self.assertTrue(any("linearized" in m for m in self._msgs(d, facts))) + + def test_minus_no_bundling(self): + facts = ootb.Facts("m", 6, set(), 100, effort="minus") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + self.assertTrue(any("minus" in m for m in self._msgs(d, facts))) + + def test_outcome_based(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + outcome = {"converged": False, "iterations": 100, "rel_gap": 0.2} + self.assertTrue(any("iterations" in m for m in self._msgs(d, facts, outcome))) + + def test_disabled_generator_skipped(self): + facts = ootb.Facts("m", 1, set(), 3, effort="base") + d = ootb.Decision(run_ef=True, ef_reason="min_ranks") + pol = ootb.load_policy() + pol["suggestions"]["disabled"] = ["_sg_ran_ef_few_ranks"] + msgs = ootb.make_suggestions(d, facts, pol) + self.assertFalse(any("only 1 MPI" in m for m in msgs)) + + +if __name__ == "__main__": + unittest.main() diff --git a/run_coverage.bash b/run_coverage.bash index 3dc8889d9..d5a81a0ae 100755 --- a/run_coverage.bash +++ b/run_coverage.bash @@ -122,6 +122,9 @@ run_phase "test_smps (serial)" \ run_phase "test_generic_cylinders (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_generic_cylinders.py -v +run_phase "test_out_of_the_box (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_out_of_the_box.py -v + run_phase "test_ootb_validate (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_validate.py -v From 3ccb1da041786366daeb062f0b8a998de53d56a0 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Wed, 1 Jul 2026 16:57:00 -0700 Subject: [PATCH 34/34] Out-of-the-box: route solver choice by problem class (LP/MIP/QP/MIQP/NLP/MINLP) The base/plus probe now classifies the model from its integrality (vars_int) and the max objective/constraint degree (new model_degree fact: linear / quadratic / more-than-quadratic) into one of six problem classes, and OOTB picks the first available solver from that class's preference list. So a model that is more nonlinear than quadratic routes to a general NLP solver (ipopt), and an integer model never routes to a continuous-only solver. - policy: preference_order_by_class with one preferred-solver list per class; NLP -> ipopt, MINLP -> baron/scip/bonmin/couenne (rarely installed -> warn), continuous QP/NLP drop cbc/glpk, integer lists drop ipopt. preference_order stays the master superset (detection + minus-tier fallback); each class list is a subset of it. - interpreter: _model_degree() detector (polynomial_degree None or > 2 => nonlinear) wired into the probe size profile; Decision.problem_class; class-routed selection with a no-solver-for-this-class suggestion. - validator: static check that all six classes exist and each is a non-empty subset of preference_order; synthetic checks that each class routes correctly and a nonlinear model with only MIP solvers picks nothing (no wrong fallback). - tests + design doc updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/out_of_the_box_design.md | 4 +- .../ootb_policies/ootb_policy_2026-06-28.json | 21 ++- mpisppy/generic/ootb_validate.py | 42 ++++++ mpisppy/generic/out_of_the_box.py | 124 ++++++++++++++++-- mpisppy/tests/test_ootb_validate.py | 9 ++ mpisppy/tests/test_out_of_the_box.py | 96 +++++++++++++- 6 files changed, 278 insertions(+), 18 deletions(-) diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md index b491e8ef8..6ffaa4ef3 100644 --- a/doc/designs/out_of_the_box_design.md +++ b/doc/designs/out_of_the_box_design.md @@ -191,7 +191,7 @@ with calibration output.) Schema keys: | Key | Purpose | |---|---| | `ef_fallback` | when to solve the EF: rank floor `min_ranks_for_decomposition` (req. 3); else the **same effort model as bundles** on the whole problem — base `effort(num_scens) ≤ ef_effort_budget`, plus measured `ef_target_seconds`, minus count `ef_if_num_scens_at_most` | -| `solver` | `preference_order` (persistent-commercial → commercial → free QP-capable → LP/MIP-only), plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | +| `solver` | `preference_order_by_class` — one preferred-solver list per **problem class** (`LP`, `MIP`, `QP`, `MIQP`, `NLP`, `MINLP`); the base/plus probe classifies the model from its integrality (`vars_int`) and max objective/constraint degree (`model_degree`: linear/quadratic/more-than-quadratic → NLP/MINLP), so a nonlinear model routes to `ipopt` and an integer model never routes to `ipopt`. `preference_order` is the master superset (detection candidates + the minus-tier fallback when the class is unknown); each class list is a subset of it. Plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | | `hub` | default hub factory (`ph_hub`, no flag) | | `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | | `rank_allocation` | small-core roster widened by ranks (add a rung only while each cylinder keeps ≥ `min_ranks_per_cylinder`, ≤ `max_cylinders`); ranks split **unbalanced** across cylinders by `rank_ratios` (xhatter 0.2) — crude cold-start (§5.5) | @@ -237,7 +237,7 @@ to the tier. | Tier | Flag | Instantiates | New facts | What it can decide (vs. advise) | |---|---|---|---|---| | minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability; **cannot bundle**. Integrality/size unknown → only **advises** on prox linearization | -| base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int` | EF gate **size-aware**; integrality **decides** ipopt/HiGHS/linearize-prox; effort-budgeted bundle sizing (§5.3) | +| base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int`, `model_degree` | EF gate **size-aware**; integrality + degree **decide** the problem class (LP/MIP/QP/MIQP/NLP/MINLP) and hence the solver (nonlinear → ipopt); linearize-prox for LP/MIP-only solvers; effort-budgeted bundle sizing (§5.3) | | plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | **`plus` is still OOTB, not a tuning tool.** It makes the *same* one-shot diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json index 4c203f58f..0374612dd 100644 --- a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -16,7 +16,7 @@ "_calibration_note": "ef_effort_budget derived from ef_target_seconds / seconds_per_effort_unit (calibrated 2026-06-28)." }, "solver": { - "_comment": "Picked by trying preference_order in order and keeping the first that SolverFactory(name).available(exception_flag=False) reports. Persistence is requested by NAMING the *_persistent interface (no separate flag). Persistent commercial first, then plain commercial, then free QP-capable, then LP/MIP-only (which forces prox linearization).", + "_comment": "The base/plus probe classifies the model into one of six problem classes (LP, MIP, QP, MIQP, NLP, MINLP) from its integrality (vars_int) and the max polynomial degree of its objective/constraints (linear / quadratic / more-than-quadratic). OOTB then picks the first AVAILABLE solver from that class's list in preference_order_by_class -- e.g. a more-than-quadratic (NLP) model routes to ipopt, an integer model never routes to ipopt. Availability is SolverFactory(name).available(exception_flag=False). Persistence is requested by NAMING the *_persistent interface (no separate flag), so persistent commercial comes first within each class. preference_order is the MASTER superset (the detection candidate set, and the fallback when the class is unknown -- the minus tier does not instantiate, so it cannot classify). Every per-class list is a subset of it (the validator enforces this). Continuous QP/NLP lists drop the LP/MIP-only solvers (cbc/glpk); integer lists drop ipopt (continuous only); MINLP lists dedicated global solvers, which are rarely installed (OOTB then warns).", "preference_order": [ "gurobi_persistent", "cplex_persistent", @@ -28,8 +28,21 @@ "highs", "ipopt", "cbc", - "glpk" + "glpk", + "baron", + "scip", + "bonmin", + "couenne" ], + "preference_order_by_class": { + "_comment": "Per-problem-class preferred solver lists (design sec. 6). Keyed by class: LP, MIP, QP, MIQP, NLP, MINLP. Each list is tried top-to-bottom; the first installed solver wins. Every entry must also appear in preference_order. Note: on the DECOMPOSITION path PH adds a quadratic proximal term, so an LP/MIP model's subproblem is really a QP/MIQP; when the chosen solver is LP/MIP-only (lp_mip_only_force_linearize_prox) OOTB linearizes that prox instead of requiring a QP-capable solver, which is why cbc/glpk remain acceptable for LP/MIP.", + "LP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "cbc", "glpk"], + "MIP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "cbc", "glpk"], + "QP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "ipopt"], + "MIQP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs"], + "NLP": ["ipopt"], + "MINLP": ["baron", "scip", "bonmin", "couenne"] + }, "commercial": [ "gurobi", "cplex", @@ -54,8 +67,8 @@ "cbc" ], "caveats": { - "_comment": "Surfaced via suggestions, not branched on yet (model integrality is not introspected in v1; see design \u00a76).", - "ipopt": "Continuous NLP only; cannot handle integer variables.", + "_comment": "Human-readable notes about individual solvers. The base/plus probe now DOES introspect integrality and nonlinearity to route solver selection (see preference_order_by_class), so these caveats mostly explain the routing; the remaining ones are surfaced via suggestions.", + "ipopt": "Continuous NLP only; cannot handle integer variables (so it is in the NLP/QP lists, never the MIP/MIQP/MINLP lists).", "highs": "QP ok; mixed-integer quadratic (MIQP) support is limited, so the PH prox on MIP subproblems may be rejected.", "glpk_cbc": "LP/MIP only; cannot handle the quadratic PH prox term, so OOTB sets --linearize-proximal-terms." } diff --git a/mpisppy/generic/ootb_validate.py b/mpisppy/generic/ootb_validate.py index e5ab0c524..91d660da4 100644 --- a/mpisppy/generic/ootb_validate.py +++ b/mpisppy/generic/ootb_validate.py @@ -200,6 +200,20 @@ def add(name, ok, detail=""): add(f"solver.{key} is a subset of preference_order", isinstance(val, list) and set(val) <= set(pref or []), f"stray: {sorted(set(val) - set(pref or []))}") + # per-class preferred solver lists: all six classes present, each a non-empty + # subset of the master preference_order (so detection always probes them). + by_class = {k: v for k, v in sp.get("preference_order_by_class", {}).items() + if not k.startswith("_")} + add("solver.preference_order_by_class covers all six problem classes", + set(by_class) == {"LP", "MIP", "QP", "MIQP", "NLP", "MINLP"}, + f"have {sorted(by_class)}") + for cls, lst in by_class.items(): + add(f"solver.preference_order_by_class.{cls} is a non-empty subset of " + f"preference_order", + isinstance(lst, list) and len(lst) > 0 + and all(isinstance(s, str) for s in lst) + and set(lst) <= set(pref or []), + f"stray: {sorted(set(lst) - set(pref or []))}") # spoke ladder ladder = policy["spoke_ladder"] @@ -404,6 +418,34 @@ def add(name, ok, detail=""): add("OOTB adds at most one rho setter", len(_rho_setter_args(d)) <= 1, f"OOTB rho setters: {_rho_setter_args(d)}") + # solver routing by problem class: the chosen solver must come from the + # matching per-class preference list (base/plus, where the class is known). + by_class = policy["solver"].get("preference_order_by_class", {}) + all_solvers = set(policy["solver"]["preference_order"]) + + def _routes(name, model_degree, vars_int, available): + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_cont=10, + vars_int=vars_int, model_degree=model_degree, + available_solvers=available), policy) + want = by_class.get(d.problem_class, []) + add(f"{name} -> solver from the {d.problem_class} list", + d.chosen_solver is not None and d.chosen_solver in want, + f"class={d.problem_class} chose={d.chosen_solver} want in {want}") + + _routes("continuous linear", "linear", 0, all_solvers) + _routes("integer linear", "linear", 5, all_solvers) + _routes("continuous quadratic", "quadratic", 0, all_solvers) + _routes("integer quadratic", "quadratic", 5, all_solvers) + # more-than-quadratic continuous routes to the NLP solver, never a MIP solver + _routes("continuous nonlinear", "nonlinear", 0, {"gurobi", "ipopt"}) + + # a nonlinear model with NO NLP solver must NOT fall back to a MIP solver + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_cont=10, vars_int=0, + model_degree="nonlinear", + available_solvers={"gurobi", "cbc"}), policy) + add("nonlinear model without an NLP solver -> no solver (not a MIP solver)", + d.chosen_solver is None, f"chose={d.chosen_solver}") + return checks diff --git a/mpisppy/generic/out_of_the_box.py b/mpisppy/generic/out_of_the_box.py index 26e426cdc..1d7b09fc6 100644 --- a/mpisppy/generic/out_of_the_box.py +++ b/mpisppy/generic/out_of_the_box.py @@ -77,6 +77,10 @@ class Facts: vars_cont: int | None = None # continuous vars per scenario nonants_total: int | None = None # first-stage (nonanticipative) vars nonants_int: int | None = None # integer nonants + # user-model nonlinearity from the probe: "linear" | "quadratic" | "nonlinear" + # (more nonlinear than quadratic, i.e. degree > 2 or non-polynomial); None at + # the minus tier (nothing is instantiated, so the class cannot be determined). + model_degree: str | None = None multistage: bool = False branching_factors: list | None = None # for the equivalent command line user_solver_name: str | None = None # name the user gave (if any) @@ -98,6 +102,7 @@ class Decision: run_ef: bool = False ef_reason: str | None = None # "min_ranks" | "few_scens" | "user" ... chosen_solver: str | None = None + problem_class: str | None = None # LP|MIP|QP|MIQP|NLP|MINLP (base/plus) num_cylinders: int = 1 # hub + spokes actually configured intra_ranks: int = 1 # widest cylinder's rank count rank_split: dict = field(default_factory=dict) # cylinder -> ranks (flex) @@ -166,22 +171,35 @@ def choose(flag: str, value: str | None, reason: str) -> bool: # --- step 1: pick the solver NAME --------------------------------------- # The flag is emitted only once the EF gate decides: --EF-solver-name for the # EF, --solver-name to decompose (mpi-sppy uses different keys). Here we just - # settle on the name. + # settle on the name -- routed by the model's PROBLEM CLASS (LP/MIP/QP/MIQP/ + # NLP/MINLP) so a more-than-quadratic model gets an NLP solver (ipopt) and an + # integer model never gets a continuous-only one. sp = policy["solver"] + d.problem_class = _problem_class(facts) + if d.problem_class is not None: + d.notes.append(f"model class: {d.problem_class} ({_class_english(facts)})") if facts.user_flags & {"--solver-name", "--EF-solver-name"} and facts.user_solver_name: d.chosen_solver = facts.user_solver_name d.notes.append(f"solver: kept user's value ({facts.user_solver_name}); " f"OOTB defers") else: - for name in sp["preference_order"]: - if name in facts.available_solvers: - d.chosen_solver = name - break + by_class = sp.get("preference_order_by_class", {}) + if d.problem_class is not None and d.problem_class in by_class: + order = by_class[d.problem_class] + why = f"first available preferred for a {d.problem_class} model" + else: + # minus tier (class unknown) or a class with no dedicated list: use + # the master preference order. + order = sp["preference_order"] + why = "first available in preference order" + d.chosen_solver = _first_available(order, facts.available_solvers) if d.chosen_solver is None: - d.notes.append("WARNING: no known solver detected; user must supply one") + d.notes.append( + f"WARNING: no installed solver for this model " + f"({d.problem_class or 'class unknown'}); tried {', '.join(order)}. " + f"User must supply one (--solver-name).") else: - d.notes.append(f"solver: {d.chosen_solver} " - f"(first available in preference order)") + d.notes.append(f"solver: {d.chosen_solver} ({why})") # --- step 2: EF gate ---------------------------------------------------- # Reuses the bundle effort() model on the WHOLE problem (all scenarios as one @@ -334,6 +352,42 @@ def _fmt_ratio(x: float) -> str: return str(int(x)) if float(x).is_integer() else str(x) +def _first_available(order, available) -> str | None: + """First solver in `order` that is installed (present in `available`).""" + for name in order: + if name in available: + return name + return None + + +# (model_degree, has_integer_vars) -> problem class label. Used to route solver +# selection to the matching preference_order_by_class list. +_PROBLEM_CLASS = { + ("linear", False): "LP", + ("quadratic", False): "QP", + ("nonlinear", False): "NLP", + ("linear", True): "MIP", + ("quadratic", True): "MIQP", + ("nonlinear", True): "MINLP", +} + + +def _problem_class(facts: Facts) -> str | None: + """The model's problem class (LP/MIP/QP/MIQP/NLP/MINLP), or None when the + model was not instantiated (minus tier) so integrality/degree are unknown. + Integrality is `vars_int > 0`; the degree is `model_degree` -- "linear", + "quadratic", or "nonlinear" (more nonlinear than quadratic).""" + if facts.model_degree is None or facts.vars_int is None: + return None + return _PROBLEM_CLASS[(facts.model_degree, facts.vars_int > 0)] + + +def _class_english(facts: Facts) -> str: + """Plain-language basis for the problem class, e.g. "integer + quadratic".""" + integer = "integer" if (facts.vars_int or 0) > 0 else "continuous" + return f"{integer} + {facts.model_degree}" + + def _effort(spb: int, facts: Facts, scaling: dict) -> float: """Modeled solve effort of a bundle of `spb` scenarios, from the probe size profile (facts.vars_cont/vars_int/nonants_int) and the policy effort_scaling @@ -405,6 +459,21 @@ def _sg_ran_ef_few_ranks(d, facts, policy, outcome): return None +def _sg_no_class_solver(d, facts, policy, outcome): + # A problem class was detected but nothing installed handles it -- e.g. an + # MINLP with no baron/scip/bonmin/couenne, or an NLP with no ipopt. (Integer + # + more-than-quadratic models route to MINLP, which rarely has an installed + # solver.) recommend() already left chosen_solver None here. + if d.chosen_solver is None and d.problem_class: + listed = policy["solver"].get("preference_order_by_class", {}) \ + .get(d.problem_class, []) + names = ", ".join(listed) if listed else "none listed" + return (f"Detected a {d.problem_class} model, but none of the solvers " + f"OOTB prefers for it ({names}) are installed; install one or " + f"pass --solver-name explicitly.") + return None + + def _sg_no_persistent_solver(d, facts, policy, outcome): s = d.chosen_solver if not d.run_ef and s is not None and not s.endswith("_persistent"): @@ -453,6 +522,7 @@ def _sg_from_outcome(d, facts, policy, outcome): # generators in priority order (lower first) SUGGESTION_GENERATORS = [ _sg_ran_ef_few_ranks, + _sg_no_class_solver, _sg_no_persistent_solver, _sg_linearized_prox, _sg_more_ranks, @@ -594,10 +664,40 @@ def _user_flags() -> set: return flags +def _model_degree(model) -> str: + """Classify the user model's nonlinearity from the probe: "linear" (every + active objective/constraint has polynomial degree <= 1), "quadratic" (max + degree == 2), or "nonlinear" (degree > 2, or non-polynomial such as + log/exp/x*y/y -- reported by Pyomo as polynomial_degree() == None). + + The probe is the RAW scenario_creator model, so the PH proximal term (a + quadratic that mpi-sppy attaches later) is absent and does not inflate the + degree -- this reflects the user's own model.""" + import pyomo.environ as pyo + max_deg = 0 + for obj in model.component_data_objects(pyo.Objective, active=True, + descend_into=True): + deg = obj.expr.polynomial_degree() if obj.expr is not None else 0 + if deg is None: + return "nonlinear" + max_deg = max(max_deg, deg) + for con in model.component_data_objects(pyo.Constraint, active=True, + descend_into=True): + body = con.body + deg = body.polynomial_degree() if body is not None else 0 + if deg is None: + return "nonlinear" + max_deg = max(max_deg, deg) + if max_deg > 2: + return "nonlinear" + return "quadratic" if max_deg == 2 else "linear" + + def _size_profile(model) -> dict: """Read a built scenario's size profile: integer vs continuous variable - counts, and nonant (first-stage) counts. Used by the base/plus probe and by - --inspect-only's instantiation check.""" + counts, nonant (first-stage) counts, and the model degree (linear/quadratic/ + nonlinear). Used by the base/plus probe and by --inspect-only's + instantiation check.""" import pyomo.environ as pyo vars_int = vars_cont = 0 for v in model.component_data_objects(pyo.Var, active=True, descend_into=True): @@ -612,7 +712,8 @@ def _size_profile(model) -> dict: if not v.is_continuous(): nonants_int += 1 return {"vars_int": vars_int, "vars_cont": vars_cont, - "nonants_total": nonants_total, "nonants_int": nonants_int} + "nonants_total": nonants_total, "nonants_int": nonants_int, + "model_degree": _model_degree(model)} def _build_probe_scenario(module, cfg): @@ -660,6 +761,7 @@ def gather_facts(module, cfg, effort: str, policy: dict) -> Facts: facts.vars_cont = profile["vars_cont"] facts.nonants_total = profile["nonants_total"] facts.nonants_int = profile["nonants_int"] + facts.model_degree = profile["model_degree"] # plus tier (PR2): instantiate all + brief timed solve for solve-time facts. return facts diff --git a/mpisppy/tests/test_ootb_validate.py b/mpisppy/tests/test_ootb_validate.py index f883aeabd..7b4572c17 100644 --- a/mpisppy/tests/test_ootb_validate.py +++ b/mpisppy/tests/test_ootb_validate.py @@ -106,6 +106,15 @@ def test_rho_setter_must_list_all_setters(self): self.policy["option_categories"]["rho_setter"]["superseded_by"] = ["--grad-rho"] self.assertTrue(self._fails(val.validate_static(self.policy))) + def test_class_list_with_stray_solver(self): + # a per-class list must be a subset of preference_order + self.policy["solver"]["preference_order_by_class"]["NLP"] = ["not_a_solver"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_missing_a_problem_class(self): + del self.policy["solver"]["preference_order_by_class"]["MINLP"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + def test_decision_layer_detects_broken_ef_gate(self): # If the rank floor is absurdly high, OOTB would never decompose; the # "large problem -> decompose" decision check must then fail. diff --git a/mpisppy/tests/test_out_of_the_box.py b/mpisppy/tests/test_out_of_the_box.py index 56a194b27..bf2269ee0 100644 --- a/mpisppy/tests/test_out_of_the_box.py +++ b/mpisppy/tests/test_out_of_the_box.py @@ -21,6 +21,8 @@ import sys import unittest +import pyomo.environ as pyo + import mpisppy.utils.config as config from mpisppy.generic import parsing from mpisppy.generic import out_of_the_box as ootb @@ -104,8 +106,10 @@ def test_verify_instantiation_and_standalone(self): cfg, module = _farmer_cfg() profile = ootb.verify_instantiation(module, cfg) self.assertEqual(set(profile), - {"vars_int", "vars_cont", "nonants_total", "nonants_int"}) + {"vars_int", "vars_cont", "nonants_total", "nonants_int", + "model_degree"}) self.assertGreater(profile["vars_cont"], 0) + self.assertEqual(profile["model_degree"], "linear") # farmer is an LP ootb.inspect_only_standalone(module, cfg) # prints, returns None @@ -223,6 +227,96 @@ def test_disabled_generator_skipped(self): msgs = ootb.make_suggestions(d, facts, pol) self.assertFalse(any("only 1 MPI" in m for m in msgs)) + def test_no_class_solver_suggestion(self): + # MINLP with nothing installed -> "install one or pass --solver-name" + facts = ootb.Facts("m", 6, {"gurobi"}, 10, effort="base", + vars_int=3, vars_cont=5, model_degree="nonlinear") + d = ootb.Decision(run_ef=False, chosen_solver=None, problem_class="MINLP") + self.assertTrue(any("MINLP" in m for m in self._msgs(d, facts))) + + +def _tiny_model(kind): + """A one-scenario-shaped model whose objective/constraint degree we control.""" + m = pyo.ConcreteModel() + m.x = pyo.Var(bounds=(0, 10)) + m.y = pyo.Var(bounds=(0, 10)) + m.c = pyo.Constraint(expr=m.x + m.y <= 5) # linear unless overridden + if kind == "linear": + m.o = pyo.Objective(expr=m.x + 2 * m.y) + elif kind == "quadratic": + m.o = pyo.Objective(expr=m.x ** 2 + m.y) + elif kind == "cubic": + m.o = pyo.Objective(expr=m.x ** 3 + m.y) + elif kind == "nonpoly": + m.o = pyo.Objective(expr=pyo.log(m.x + 1) + m.y) + elif kind == "quad_constraint": + m.o = pyo.Objective(expr=m.x + m.y) + m.c2 = pyo.Constraint(expr=m.x * m.y <= 4) # degree comes from a con + return m + + +class TestModelDegreeAndClass(unittest.TestCase): + def test_model_degree(self): + self.assertEqual(ootb._model_degree(_tiny_model("linear")), "linear") + self.assertEqual(ootb._model_degree(_tiny_model("quadratic")), "quadratic") + self.assertEqual(ootb._model_degree(_tiny_model("cubic")), "nonlinear") + self.assertEqual(ootb._model_degree(_tiny_model("nonpoly")), "nonlinear") + # a quadratic CONSTRAINT (linear objective) still makes the model QP + self.assertEqual(ootb._model_degree(_tiny_model("quad_constraint")), + "quadratic") + + def test_problem_class_mapping(self): + def pc(degree, vint): + return ootb._problem_class( + ootb.Facts("m", 3, set(), 10, vars_int=vint, model_degree=degree)) + self.assertEqual(pc("linear", 0), "LP") + self.assertEqual(pc("quadratic", 0), "QP") + self.assertEqual(pc("nonlinear", 0), "NLP") + self.assertEqual(pc("linear", 5), "MIP") + self.assertEqual(pc("quadratic", 5), "MIQP") + self.assertEqual(pc("nonlinear", 5), "MINLP") + # minus tier: not instantiated -> class unknown + self.assertIsNone(ootb._problem_class(ootb.Facts("m", 3, set(), 10))) + + +class TestSolverRoutingByClass(unittest.TestCase): + def setUp(self): + self.policy = ootb.load_policy() + + def _rec(self, degree, vars_int, available): + facts = ootb.Facts("m", 6, set(available), 10, effort="base", + vars_int=vars_int, vars_cont=5, model_degree=degree) + return ootb.recommend(facts, self.policy) + + def test_nonlinear_continuous_routes_to_ipopt(self): + d = self._rec("nonlinear", 0, {"gurobi", "ipopt"}) + self.assertEqual(d.problem_class, "NLP") + self.assertEqual(d.chosen_solver, "ipopt") # not gurobi + + def test_nonlinear_without_nlp_solver_picks_nothing(self): + # a MIP solver must NOT be chosen for a nonlinear model + d = self._rec("nonlinear", 0, {"gurobi", "cbc"}) + self.assertEqual(d.problem_class, "NLP") + self.assertIsNone(d.chosen_solver) + + def test_integer_model_never_routes_to_ipopt(self): + d = self._rec("linear", 5, {"ipopt", "cbc"}) + self.assertEqual(d.problem_class, "MIP") + self.assertEqual(d.chosen_solver, "cbc") # ipopt can't do integers + + def test_integer_nonlinear_is_minlp(self): + d = self._rec("nonlinear", 3, {"gurobi", "ipopt"}) + self.assertEqual(d.problem_class, "MINLP") + self.assertIsNone(d.chosen_solver) # no MINLP solver installed + + def test_user_solver_wins_over_class_routing(self): + facts = ootb.Facts("m", 6, {"gurobi", "ipopt"}, 10, effort="base", + vars_int=0, vars_cont=5, model_degree="nonlinear", + user_solver_name="cbc", + user_flags={"--solver-name"}) + d = ootb.recommend(facts, self.policy) + self.assertEqual(d.chosen_solver, "cbc") # OOTB defers to the user + if __name__ == "__main__": unittest.main()