diff --git a/.github/workflows/test_pr_and_main.yml b/.github/workflows/test_pr_and_main.yml index 7d5532770..1d797377d 100644 --- a/.github/workflows/test_pr_and_main.yml +++ b/.github/workflows/test_pr_and_main.yml @@ -124,6 +124,18 @@ jobs: run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_generic_cylinders.py -v + - name: Test out-of-the-box interpreter wiring + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_out_of_the_box.py -v + + - name: Test out-of-the-box policy validator (layers 1 + 2-synthetic) + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_validate.py -v + + - name: Test out-of-the-box effort calibrator (pure fit) + run: | + coverage run $COV_ARGS -m pytest mpisppy/tests/test_ootb_calibrate.py -v + - name: Test xhat from file run: | coverage run $COV_ARGS -m pytest mpisppy/tests/test_xhat_from_file.py -v diff --git a/doc/designs/out_of_the_box_design.md b/doc/designs/out_of_the_box_design.md new file mode 100644 index 000000000..6ffaa4ef3 --- /dev/null +++ b/doc/designs/out_of_the_box_design.md @@ -0,0 +1,577 @@ +# Out-of-the-box auto-configuration — design + +**Status:** Design phase. Branch `outOfTheBox` (off Pyomo/mpi-sppy `main`), on +the DLWoodruff fork. Proceeding deliberately ("slowly"): requirements confirmed +and the major design questions resolved — decision-logic mechanism (§5), policy +file + path selection (§5.1), effort tiers (§5.2), bundle sizing & the EF gate +(§5.3), `--inspect-only` (§5.4), rank allocation (§5.5), and the PR1 tooling — +validator (§8) and effort calibrator (§9). The first dated policy file is +committed, as is an interpreter *sketch* (§7) whose decision logic is complete +with the environment wiring stubbed. Next: turn the sketch into PR1 code. +**Author:** dlw (captured with Claude Code assistance) +**Last updated:** 2026-06-28 + +--- + +## 0. Vocabulary + +**Out-of-the-box (OOTB)** mode: a CLI switch (`--out-of-the-box`, with a lighter +`--out-of-the-box-minus` and a heavier `--out-of-the-box-plus` variant — §5.2) +that lets a relatively naive user obtain a *sensible* mpi-sppy run with almost +no knowledge of the library's internals. The user supplies a model module (and +scenario data); mpi-sppy **introspects the environment and the model** and +**auto-assembles a defensible configuration** — algorithm, spokes, bundling, +solver — rather than requiring a hand-crafted hub/spoke command line. + +The *spirit* is to lower the barrier to entry: the newcomer effectively says +"here is my model, go," and gets a reasonable decomposition plus a clear +explanation of what was chosen and how to do better. + +--- + +## 1. Goals and non-goals + +### Goals + +1. A `--out-of-the-box` option (CLI entry `generic_cylinders.py`, implemented + in the refactored `mpisppy/generic/` package) that sets run options + automatically. +2. **User options always win** (requirement 0). OOTB only *fills gaps*; any + option the user explicitly set is retained verbatim and never overridden. +3. **Environment + model introspection** (requirement 2). At minimum: the + model module, the MPI rank count, and which solvers are actually installed. + Where the OS / SLURM permits: core count and memory. Plus any options the + user already supplied. +4. **Floor at 3 ranks** (requirement 3): with fewer than 3 ranks there is no + useful cylinder configuration (hub + >= 2 spokes), so OOTB solves the **EF** + instead. +5. **Transparency** (requirement 4): print (a) the **equivalent explicit + command line** the choices imply, so the user can reproduce / learn from / + tweak it, and (b) a short, prioritized list of **suggestions** (labelled + "Suggestions"), **written after the run executes** so it can also reflect + how the run went (e.g., a persistent solver, more ranks). The run proceeds + regardless. +6. **Proper bundling is central** (requirement 5): auto-forming proper bundles + from scenario count vs. available ranks is a first-class part of the + decision, not an afterthought. +7. **Quick-start documentation** (requirement 1): OOTB becomes the + recommended on-ramp in the quick start. + +### Non-goals (initial cut) + +- Replacing the explicit cylinder command line for expert users. OOTB is an + *on-ramp*, and it deliberately emits the explicit command line it chose. +- Tuning convergence parameters to optimality. OOTB aims for *defensible*, + not *optimal*, configurations — this includes `--out-of-the-box-plus`, which is + OOTB *with more information*, not an autotuner (§5.2). +- Running a solver to make decisions in the default path. Trial solves are + confined to the opt-in `plus` tier (§5.2); minus and base never solve. + +--- + +## 2. Confirmed scope decisions + +These were raised as scoping questions and confirmed by the user (2026-06-28): + +1. **Home + scope.** `generic_cylinders.py` is the primary (initial) entry + point. Reachability via `Amalgamator` is possible later but not the first + target. *(CONFIRMED as the assumed direction; revisit if Amalgamator + coverage is wanted sooner.)* +2. **What we read from the module.** OOTB learns scenario count / names (hence + two-stage vs. multistage structure) from the module. Whether OOTB also + *instantiates scenarios* to gauge size/difficulty is **resolved** as a + user-selectable effort axis (none / one probe / all) — see §5.2. +3. **"Dated data files."** Interpreted as a versioned, **date-stamped + knowledge base** of heuristics/benchmarks ("for problems like X with + resources like Y, configuration Z worked well") that *guides* the chooser + and can be refreshed over time — **not** user run-config files. +4. **"What would help" output.** Echo the equivalent command line *and* a + short prioritized **Suggestions** list — the command line up front, the + suggestions after the run; the run proceeds anyway. + +--- + +## 3. Inputs OOTB consults + +| Source | Items | Notes | +|---|---|---| +| User-supplied options | anything already on the command line / Config | Highest precedence; never overridden | +| Model module | scenario count/names, stage structure (2-stage vs multistage); at base/plus also integrality, per-scenario size, nonant count | Depth set by the effort tier (§5.2): minus structural-only, base one probe, plus all | +| MPI environment | number of ranks | Drives EF-vs-cylinders and the 3-rank floor | +| Installed solvers | which solvers import / are licensed; persistent variants | Drives solver choice + a "get a persistent solver" hint | +| OS / SLURM (best effort) | core count, memory | OS-dependent; SLURM env vars when present | + +--- + +## 4. Outputs OOTB produces + +1. A fully-populated `Config` (or equivalent) that the normal driver path then + executes. +2. A printed **equivalent explicit command line** (up front, before the run). +3. The run itself (EF when the EF gate trips — too few ranks, or the problem is + small enough per §5.1/§5.3 — otherwise the chosen cylinder configuration); + **skipped entirely under `--inspect-only`** (§5.4). +4. A printed prioritized **Suggestions** list, emitted **after the run** + (labelled "Suggestions"). Mostly **computed** from live facts / decision / + run outcome, not canned (§5.1). + +--- + +## 5. Decision-logic mechanism (RESOLVED 2026-06-28) + +The first design question — expert system vs. neural net vs. nested case/ifs, +all optionally driven by dated data files — is **resolved**: + +**Authored, declarative, data-driven: a dated/versioned policy file holding the +knowledge, interpreted by a thin hand-written Python decision routine. No +rules-engine library, no neural net.** + +Reasoning: + +- The three candidates are really two categories: *authored* knowledge + (expert system, nested ifs — the same thing at two points on a spectrum) vs. + *learned* knowledge (neural net). The "dated data files" idea cuts across + all three and is the real commitment: **knowledge lives in data, not frozen + in code.** +- **Neural net rejected** on four independent grounds: cold start (no training + corpus, and never NN-scale data), the hard transparency requirement (must + explain *why* — req. 4), the contributor base (optimization researchers edit + rules, not models), and testability. +- **Rules-engine libraries rejected** (pyke is abandonware ~2010; experta is + stale ~2018 with `frozendict` pin problems; clipspy/CLIPS is maintained but + heaviest — C extension + its own language — for our smallest need). A RETE + engine earns its cost only with many interacting, re-firing rules; OOTB makes + a handful of decisions once, in an obvious order. An engine also adds a heavy + dependency into already-fragile MPI/solver installs and can make firing-order + *harder* to explain. +- So: implement the "expert-system framing" (facts + declarative knowledge + + thin matcher) as a thin, hand-written Python interpreter reading a JSON policy + file. Conditions are **plain coded checks** over the facts — no expression + language in v1. + +**Migration path preserved:** when a benchmark corpus eventually exists it +*tunes the numbers inside the next dated policy file* (e.g. a regression that +sets the EF cutoff or bundle target); structure stays rule-based and +explainable. **Escape hatch:** if the knowledge base ever explodes into dozens +of forward-chaining rules, revisit **clipspy** (never experta/pyke) by porting +the policy file's contents into a CLIPS KB — the policy-as-data design keeps +that open. We expect never to need it. + +Criteria this satisfies: transparency (every decision logs its reason → +equivalent command line + suggestions fall straight out), cold-start (authored v1 +works day one), maintainability (Python + JSON), testability (deterministic +"facts X ⇒ config Y" unit tests). + +## 5.1 Policy file: location, selection, and v1 schema + +**Location.** Dated policy files ship with the library under +`mpisppy/generic/ootb_policies/` (co-located with the OOTB code, which lives in +the refactored `mpisppy/generic/` package — `parsing.py`, `ef.py`, `hub.py`, +`spokes.py`, `scenario_io.py`, … ; `generic_cylinders.py` remains the +user-facing CLI entry that delegates into this package). First file: +`ootb_policies/ootb_policy_2026-06-28.json`. + +**Selection.** Each OOTB flag takes an **optional policy-file path** (see the +§5.2 mechanism): a bare flag uses the shipped default (the newest dated file +whose name carries no focus token, e.g. `ootb_policy_.json`); +`--out-of-the-box PATH` uses `PATH`. There is **no separate policy-file flag** — +the optional value *is* the path. Multiple policy files with **different foci** +may ship side by side; the **focus is conveyed by the filename** (e.g. +`ootb_policy_quick_.json`), which is human-facing documentation only — the +program does not parse focus from the name, and no machine-read `focus` field is +needed. A user selects a focus simply by passing that file's path. The run +**logs which policy file (and `policy_version`) it used**, for reproducibility. + +**v1 schema** (see the file for the authoritative, self-documenting copy). The +*structure* is authored; the *numbers* are produced by the calibration tool (§9) +on the example set — reproducible, not hand-guesses. (The current design-phase +file still carries hand-guesses flagged `_cold_start_guess`; PR1 replaces them +with calibration output.) Schema keys: + +| Key | Purpose | +|---|---| +| `ef_fallback` | when to solve the EF: rank floor `min_ranks_for_decomposition` (req. 3); else the **same effort model as bundles** on the whole problem — base `effort(num_scens) ≤ ef_effort_budget`, plus measured `ef_target_seconds`, minus count `ef_if_num_scens_at_most` | +| `solver` | `preference_order_by_class` — one preferred-solver list per **problem class** (`LP`, `MIP`, `QP`, `MIQP`, `NLP`, `MINLP`); the base/plus probe classifies the model from its integrality (`vars_int`) and max objective/constraint degree (`model_degree`: linear/quadratic/more-than-quadratic → NLP/MINLP), so a nonlinear model routes to `ipopt` and an integer model never routes to `ipopt`. `preference_order` is the master superset (detection candidates + the minus-tier fallback when the class is unknown); each class list is a subset of it. Plus `commercial` / `qp_capable` / `lp_mip_only_force_linearize_prox` sets and `caveats` | +| `hub` | default hub factory (`ph_hub`, no flag) | +| `spoke_ladder` | ordered `rungs` of WIRED spoke flags (outer/inner), `core_roster_min` (≥1 outer + ≥1 inner = the 3-rank floor), `max_cylinders` | +| `rank_allocation` | small-core roster widened by ranks (add a rung only while each cylinder keeps ≥ `min_ranks_per_cylinder`, ≤ `max_cylinders`); ranks split **unbalanced** across cylinders by `rank_ratios` (xhatter 0.2) — crude cold-start (§5.5) | +| `effort_scaling` | shape of solve effort vs. size (continuous ~linear, integers superlinear via `int_exponent`); shared by bundle sizing (§5.3) | +| `bundle_sizing` | how big bundles are (base/plus only — **minus cannot bundle**): largest `spb` within an effort budget (base = relative M; plus = measured seconds); `--scenarios-per-bundle` divides `num_scens`; `#bundles ≥ #ranks` | +| `option_categories` | per-concern default options (`rho_setter` `--grad-rho`, `termination` `--rel-gap 0.01`, `max_iterations` `--max-iterations 100`, `dynamic_rho` `--dynamic-rho-primal-crit`), each skipped if the user set any flag in its `superseded_by` list; skipped on the EF path | +| `additional_options` | catch-all for other extra flags (each with an optional `superseded_by`, default = its own flag) | +| `suggestions` | toggles/tunes the **computed** suggestion generators (`disabled` suppresses specific ones); the prose lives in code, emitted after the run | + +**Additional options, with per-concern override.** Beyond the structural +choices, the policy applies extra options grouped by **concern** in +`option_categories` — `rho_setter` (`--grad-rho`), `termination` +(`--rel-gap 0.01`), `max_iterations` (`--max-iterations 100`), `dynamic_rho` +(`--dynamic-rho-primal-crit`). Each carries a **`superseded_by`** list of user +flags that obviate OOTB's default for that concern, so OOTB backs off when the +user addresses the concern with *any* equivalent flag — not just the identical +one. This matters concretely: mpi-sppy allows **only one rho setter active**, so +adding `--grad-rho` when the user chose `--sensi-rho` would be a *hard error*, so +`rho_setter.superseded_by` lists all the rho setters. A leftover +`additional_options` catch-all handles miscellaneous flags (each with an optional +`superseded_by` that defaults to its own flag). Conditionality across problem +*classes* is still expressed by **focus** (which file ships which options). All +are decomposition-run options, skipped on the EF path. (`--dynamic-rho-primal-crit` +is a boolean — no argument; its threshold `dynamic_rho_primal_thresh` defaults to +0.1 — and needs an active rho setter, which OOTB's default `--grad-rho` provides.) + +**Suggestions are computed, not canned.** The post-run **Suggestions** (req. 4, +§4) are produced by small Python *generators* that compute text from live +facts / decision / run-outcome; the policy only toggles/tunes them +(`suggestions.disabled`). This keeps the split clean: **decisions stay +data-driven (this policy); suggestions are the computed diagnostics layer.** + +## 5.2 Effort tiers (instantiation depth) — RESOLVED 2026-06-28 + +OOTB exposes three tiers along an **effort axis** — how deeply it inspects the +model — selected by mutually-exclusive flags. They share ONE interpreter and +ONE policy file; the tiers differ only in how deeply `gather_facts()` populates +the `Facts` object. Every decision uses the **best fact available** and +**degrades gracefully** (to structural reasoning, or to a suggestion) when a +fact is absent. Requirement 0 (the user's explicit options win) is orthogonal +to the tier. + +| Tier | Flag | Instantiates | New facts | What it can decide (vs. advise) | +|---|---|---|---|---| +| minus | `--out-of-the-box-minus` | nothing | scenario count, ranks, solvers, stage structure | EF gate by **count**; solver by availability; **cannot bundle**. Integrality/size unknown → only **advises** on prox linearization | +| base (default) | `--out-of-the-box` | **one** probe scenario | size profile: `vars_int`, `vars_cont`, `nonants_total`, `nonants_int`, `model_degree` | EF gate **size-aware**; integrality + degree **decide** the problem class (LP/MIP/QP/MIQP/NLP/MINLP) and hence the solver (nonlinear → ipopt); linearize-prox for LP/MIP-only solvers; effort-budgeted bundle sizing (§5.3) | +| plus (later) | `--out-of-the-box-plus` | **all** + brief solve | per-subproblem solve time, LP-relax / integrality gap | iteration/time-limit defaults, bundle sizing to amortize solve cost, "hard MIP" signals | + +**`plus` is still OOTB, not a tuning tool.** It makes the *same* one-shot +decisions as base — only with a richer fact base (measured single-scenario solve +time, LP-relaxation / integrality gap). It does **not** iterate, search a +parameter space, or auto-tune to optimality (a non-goal, §1); think *"out-of-the- +box with more information,"* not an autotuner. (Offline coefficient fitting is a +separate concern — the calibration tool, §9.) + +**Mechanism.** The three flags set one internal `ootb_effort` level +(`minus`/`base`/`plus`); at most one may be supplied. Each takes an **optional +policy-file path** — declared `domain=str, default=None, +argparse_args={'nargs': '?', 'const': }` — giving three states: +absent → `None` → OOTB off; bare flag → `const` → the default policy; `PATH` → +that policy file. (A `bool` domain cannot be used: pyomo's +`declare_as_argument` forces `store_true` for bool, which takes no value; +`str` + `nargs='?'` is the supported optional-value form, and `add_to_config` +forwards `argparse_args` straight to `argparse.add_argument`.) This keeps a +single code path — richer tiers merely turn suggestions into decisions. + +**Why a probe, not reuse-all (the intervention seam).** The *structural* +choices — EF-vs-decomposition, bundling, cylinder/comm/rank layout — must be +fixed *before* models are built: they determine what is instantiated and the +MPI topology, and proper bundling builds a *new combined* Pyomo model, so +singleton instantiations cannot be reused as bundles (Pyomo components cannot +be re-parented). Instantiating everything first and then restructuring wastes +work and reshuffles comms. So **base** instantiates only a cheap *probe* +(discarded) for the structural decisions; the normal driver then does the real, +layout-correct instantiation — needed anyway — which also feeds any *parametric* +refinement (solver, spokes, rho, iterations) where intervening is free. +**plus** accepts more redundant work by design, in exchange for solve-time / +gap information. + +**`probe_scenarios` knob (lands with the base tier).** How many representative +scenarios base/plus instantiate (`scenario_names[:probe_scenarios]`). Default +**1** — enough for structurally homogeneous scenarios; bump it for models whose +size or integrality varies by scenario. Added to the policy file when base is +implemented (PR1); not in the 2026-06-28 file. + +## 5.3 Bundle sizing & effort scaling — how big, not just whether + +The hard question is not *whether* to bundle but *how big* bundles should be — +unportable in raw variable counts (the same count is trivial for one model, +intractable for another). The design makes bundle size a **derived** quantity: +pick the **largest** `scenarios_per_bundle` (`spb`) that (a) divides +`num_scens`, (b) leaves at least `B_min = max(intra_ranks, +min_bundles_per_intra_rank · intra_ranks)` bundles, and (c) keeps a bundle's +**modeled solve effort** within a budget. + +**Shared effort shape (`effort_scaling`).** One policy block models how solve +effort grows with sub-problem size, from the probe profile (`vars_cont`, +`vars_int`, `nonants_int`): + +> `effort(spb) = cont_coeff·(spb·vars_cont) + int_weight·(spb·vars_int)^int_exponent + int_nonant_coeff·nonants_int` + +Continuous content is ~linear; integers are **superlinear** (`int_exponent > 1` +captures branch-and-bound blow-up); integer nonants are a fixed per-bundle +coupling cost. (Second-stage integers scale with `spb`; integer *nonants* are +first-stage, shared once per bundle — hence a fixed term, not a `spb` +multiplier.) + +**Two anchors, one shape.** Both tiers call the same `effort(spb)`; only the +budget differs: + +- **base** (relative, no measurement): accept the largest `spb` with + `effort(spb)/effort(1) ≤ base_max_hardness_vs_single_scenario` (**M**). M is + unit-free and portable — "a bundle may be at most M× as hard as one + scenario." Pure-continuous ⇒ allowed `spb ≈ M`; pure-integer ⇒ + `≈ M^(1/int_exponent)`, automatically smaller. +- **plus** (absolute, measured): measure `t₁` = a single-scenario solve (capped + at `plus_probe_solve_time_cap_seconds`), predict + `t(spb) ≈ t₁·effort(spb)/effort(1)`, accept the largest `spb` with + `t(spb) ≤ plus_target_seconds_per_bundle`. +- **minus** (no profile): **cannot bundle** — with no model information there is + no safe way to size a bundle, so minus always runs unbundled. + +**Measurement does not remove the JSON assumptions.** `plus` only pins the +*scale* (`t₁`); the *shape* (`int_exponent`, weights) still comes from +`effort_scaling`. And MIP solve times are **noisy and non-monotone** (a bigger +MIP can solve faster), so a single timing must not drive the whole choice — the +JSON shape is a **prior/regularizer** that measurement calibrates. (Later +refinement: `plus` measures two points to nudge `int_exponent` locally.) + +**EF gate uses the same effort model.** The EF is just all scenarios as one +model, so the EF gate reuses `effort()` on the whole problem: above the rank +floor, run the EF when `effort(num_scens)` is within an **EF budget**. Unlike +bundle sizing's *relative* budget (M× a single scenario), the EF budget is +**absolute** — the monolith has no single-scenario reference: **base** +`effort(num_scens) ≤ ef_effort_budget` (same effort units as bundle effort); +**plus** measured `t₁·effort(num_scens)/effort(1) ≤ ef_target_seconds`; **minus** +(no profile) falls back to the count rule `num_scens ≤ ef_if_num_scens_at_most`. +Because the units match bundle effort, the EF budget and bundle budgets are +mutually consistent: when the whole problem exceeds the EF budget, OOTB +decomposes and sizes each bundle within its own (smaller) effort budget. + +**User-forced decomposition overrides the gate.** If the user explicitly set any +**decomposition flag** — a wired spoke or a non-default hub (`DECOMPOSITION_FLAGS` +in the interpreter) — and has at least the rank floor, OOTB **never** substitutes +the EF, even for a small problem (requirement 0). The rank floor is checked +first: below it the decomposition can't fit, so the EF is used regardless. (This +flag vocabulary is a *fact* about `generic_cylinders`, not a focus preference, so +it lives in code, not the policy; the validator checks it against the real CLI.) + +**Status.** All `effort_scaling` / `bundle_sizing` numbers are +`_cold_start_guess`es; **foci** ship different shapes (a `mip-heavy` file with a +steeper `int_exponent`), and the dated-file migration path (§5) refines the +coefficients from benchmark data. The interpreter sketch implements the base +relative sizer (`_effort`, `_pick_spb_by_effort`); minus does not bundle; the +`plus` measure-and-scale hook is stubbed. + +## 5.4 `--inspect-only` (dry run; shares OOTB's instantiation) — RESOLVED 2026-06-28 + +A general driver flag (not OOTB-specific, but documented here because it shares +code): do the inspection, **print the configuration + equivalent command line + +config-time suggestions, then stop before the production optimization run.** + +- **Semantics:** "no *production* run," not "no solver call ever." Per the + user's decision, **`--out-of-the-box-plus`'s brief calibration solves count as + inspection** (resolution B) — they are bounded by + `plus_probe_solve_time_cap_seconds` and are *how* `plus` forms its + recommendation — so `plus` + `--inspect-only` still measures, then stops. +- **Standalone (no `--out-of-the-box`):** `--inspect-only` **verifies a scenario + can be instantiated** (builds one and reports) — a cheap model smoke-test — + **reusing OOTB's probe instantiation code** (`verify_instantiation`, shared + with the base/plus probe). +- **× `minus`:** silly but allowed. `minus` says "instantiate nothing," yet + `--inspect-only` must build one scenario to verify — so **`--inspect-only` + takes priority**: a single verification instantiation happens, while the + *decision* stays minus-level (structural, no size profile fed into choices). +- **Suggestions:** only the config-time ones (nothing ran to yield + outcome-based ones). +- **Optional assumed rank count (HPC planning):** `--inspect-only N` plans as if + `N` ranks were available — so a supercomputer user can get the recommended + command line for, say, a 512-rank job *from a login node, without launching + it*. Bare `--inspect-only` uses the actually-detected ranks. Everything else + (solvers, model size) still comes from the real (possibly small) session; only + the rank count is hypothetical, and the emitted `mpiexec -np N …` reflects it. + +Ships in **PR1**: an **optional-value** flag (`domain=str, nargs='?'`, the value +being the assumed rank count) — *not* `store_true`, since it now takes a value +(the same bool-domain caveat as the effort flags). The driver short-circuits +after printing, before apply-to-`Config` / run. + +## 5.5 Rank allocation — small core, widened, and unbalanced + +Two parts (policy `rank_allocation`): + +**Roster size — prefer width over weak spokes.** Start from the minimal core +(≥1 outer + ≥1 inner spoke = hub + `--lagrangian` + `--xhatshuffle`). Add +further ladder rungs only while each cylinder would still keep +`min_ranks_per_cylinder` ranks (a coarse uniform gate), up to `max_cylinders` +(now **7**, so all six ladder rungs are reachable at enough ranks — no dead +rung). So **6 ranks → hub + lagrangian + xhatshuffle (3 cylinders), widened** — +*not* 6 single-rank cylinders. Extra ranks buy subproblem throughput, which (so +far in practice) beats piling on lower-value bound spokes. + +**Unbalanced distribution (flex-ranks).** Ranks are split across the chosen +cylinders by per-cylinder **`rank_ratios`**, *not* uniformly: cheaper cylinders +get a smaller share. v1 ships `--xhatshuffle` (and the xhat family) at **0.2**; +everything else uses `default_rank_ratio` 1.0. Ratios are normalized over the +chosen cylinders and floored at 1 rank each (e.g. 6 ranks → hub 3, lagrangian 2, +xhatshuffle 1). **This is a crude cold-start:** the right split is a much more +complicated calculation that depends on the *nature of the subproblems* +(relative solve cost), and is a natural place for the `plus` tier's measurements +to inform. The widest cylinder's rank count governs the bundling +`#bundles ≥ #ranks` floor. + +--- + +## 6. Open details + +- **Instantiation depth — RESOLVED** as the effort tiers (§5.2): minus (none), + base (one probe, default), plus (all + brief solve, later). +- **Bundle sizing — RESOLVED** as an effort-budgeted rule (§5.3): policy + `effort_scaling` shape + `bundle_sizing` budgets; interpreter `_effort` / + `_pick_spb_by_effort` (base, relative M); minus does not bundle; stubbed + `plus` measure-and-scale. Numbers are `_cold_start_guess`es. +- **EF gate — RESOLVED** (§5.3): reuses the bundle `effort()` model on the whole + problem against an absolute **EF budget** — base `ef_effort_budget`, plus + `ef_target_seconds`, minus the count rule `ef_if_num_scens_at_most`. +- **Dated data files — generation mechanism is the effort-calibration tool + (§9)**, in PR1 (calibrated on the examples); a broader benchmark corpus, + versioning, and shipping cadence are **still open**. +- **Still open:** Amalgamator reachability (§2.1) — `generic_cylinders` first. + +--- + +## 7. Phased rollout + +Per project convention, ship as review-sized phases, each green on its own. A +sketch of the interpreter already exists at `mpisppy/generic/out_of_the_box.py` +— the pure `recommend(facts, policy) → Decision` logic is complete and +smoke-tested; environment/model probing and apply-to-`Config` are stubbed. + +- **PR1 — interpreter pipeline + `--out-of-the-box-minus` + `--out-of-the-box` + (base).** These share everything except one probe instantiation, so they land + together: fact-gathering (structural + one-scenario probe), the policy + interpreter, EF-vs-cylinder / bundling / spoke selection, reporting + (equivalent command line + post-run suggestions), apply-to-`Config`, the + `probe_scenarios` knob, the `--inspect-only` dry run (§5.4, incl. the shared + `verify_instantiation`), the **policy-file validator (§8)** with its CI-gating + layers wired into CI, the **effort-calibration tool (§9)** — whose output *is* + the shipped policy's effort numbers (calibrated on the examples, not + hand-guesses) — quick-start docs, and tests. Ships the default on-ramp and the + no-instantiation escape hatch in one **large but cohesive** PR: interpreter + + validator + calibrator together make the shipped policy both *runnable* and + *assessable*. +- **PR2 (later) — `--out-of-the-box-plus`.** Full instantiation + a brief timed + solve, a probe-time budget, and handling for "doesn't solve quickly"; feeds + iteration/time-limit defaults and solve-cost-aware bundling. + +--- + +## 8. Policy-file validator (PR1 deliverable) + +A **fully automated** validator that, given a policy file, checks it is +well-formed and that its recommended configurations **run** — and **flags the +runs that look problematic** for a human to review — using the mpi-sppy +**examples** as test models. It does **not** assert correctness or "expected +results" (we have no cheap, reliable oracle for that); the reader judges whether +performance is acceptable, which is why the report is human-readable. **It ships +with PR1** for two reasons: (1) a dated policy file should not be shipped without +the tool that validates it; and (2) it is the **easiest way to *try OOTB out*** +— running it on the bundled examples exercises the whole pipeline end-to-end +(decisions, real runs, the report), so it doubles as the demonstration harness. +Because of (2), the **full tool — all three layers, including the run-tier — is +in PR1** (not a fast-follow); the small example set keeps the run-tier feasible +with a modest/free solver. Three layers, fast to slow: + +**1. Static (schema) checks.** JSON parses; required keys/types present; every +referenced flag is real — `spoke_ladder` rungs are wired spokes, solver names +known, each `option_categories[*].flag` and every `superseded_by` entry is a +valid option, `DECOMPOSITION_FLAGS` match the `generic_cylinders` vocabulary; +`_cold_start_guess` entries name real keys; numbers in range. + +**2. Decision checks (fast; `recommend()` only, no solves).** Run `recommend()` +on **hand-built synthetic `Facts`** (no instantiation — the CI-gating subset) +and, more thoroughly, on **real example models** (farmer, aircond, sizes, …; +probe-instantiated, hence out of CI) under synthetic environments (varying ranks, +solvers, problem sizes), and assert: + +- **EF invoked when it should be:** small problem or `< min_ranks` ⇒ EF. +- **EF *not* invoked when it shouldn't be:** large / integer-heavy ⇒ decompose. +- **User-forced decomposition wins:** simulated user `--ph --lagrangian + --xhatshuffle` with ≥ rank floor ⇒ never EF (§5.3). +- **Bundling validity:** when bundling, `scenarios_per_bundle` divides + `num_scens` and `#bundles ≥ #ranks`. +- **No conflicting options:** `superseded_by` simulation ⇒ OOTB never stacks a + second rho setter (which would be a hard error). + +**3. Run checks (slow; actually execute — *longer runs*).** The decision checks +only verify *what OOTB chooses*; this layer actually **runs** the recommended +configurations on the (small) examples and records what happened. We can't assert +"expected results," but two failure modes are worth **flagging automatically**: + +- **EF recommended but slow:** when OOTB recommends the EF, run it and **flag any + that fail to reach a 1% MIP gap within ten minutes** — a sign the EF was the + wrong call for that problem. +- **Cylinders maxed out on iterations:** run the decomposition — including + **forced** cylinders where the EF gate would otherwise have picked the EF, to + exercise that path — and **flag any that terminate on `max_iterations`** rather + than converging. + +Everything else (objective, bound/gap, iteration count, wall time per run) is +**recorded for the reader**, not auto-judged. The 1%-gap and ten-minute +thresholds are validator settings, not policy. These runs are **not cheap** — PH +to convergence takes time, and they need a real solver — so layer 3 runs +**nightly / on demand / locally**, never as a per-PR gate. + +**Report.** The validator produces a **report** that details **every** test, not +just failures: which example and synthetic environment, and for layer-3 runs the +objective, bound/gap, **iteration count**, and wall time. It does **not** print +"expected vs. actual" — there is no oracle — so the reader verifies that +performance is acceptable, which is exactly why the report must be +**human-readable** (a summary) as well as **machine-readable**. It **prominently +highlights** the two automatic flags: EF runs that **missed a 1% gap in ten +minutes**, and decomposition runs that **maxed out on iterations**. It names the +policy file and `policy_version` validated. + +**Only a small part can gate CI.** CI runners typically have **no commercial +solver** and tight time budgets, so the per-PR gate is limited to the cheap, +solver-free checks: layer 1 (static schema) plus the **synthetic-facts** subset +of layer 2 (pure `recommend()` decisions on hand-built `Facts`, no instantiation, +no solver). Everything else — example instantiation, anything needing a solver, +and all of layer 3 — runs **nightly / on demand / locally**, not as a gate. + +**Concrete shape (built in PR1).** A runnable module +`mpisppy/generic/ootb_validate.py` — `python -m mpisppy.generic.ootb_validate +` — exercising a fixed small example set: **farmer** (two-stage, +continuous), a small **MIP** example (e.g. `sizes`/`sslp`), and **aircond** +(multistage). Decision checks sweep a handful of synthetic +`(ranks, solvers, num_scens, size/integrality)` tuples chosen to hit each branch +(EF-small, EF-few-ranks, decompose-large, forced-decomp, bundling on/off, LP-only +solver, no-persistent). The CI gate is a **pytest** that runs only layers 1 + +2-synthetic on the shipped policy file(s) — and, per project convention, is wired +into `run_coverage.bash` **and** `test_pr_and_main.yml` in the same commit. Layer +3 is the same module invoked with `--run` (nightly / local). The report is +written human-readable and machine-readable (JSON). + +**Status:** design nailed down; **scheduled for PR1**. The check *catalog* will +keep growing, but the tool, the three-layer structure, the CI gate, and the +report all ship in PR1. + +--- + +## 9. Effort-calibration tool (future; data-tuning side) + +The `effort_scaling` coefficients and the effort budgets are abstract by +construction (arbitrary "effort units"); left as hand-guesses they are +**unassessable** — reviewers would be reviewing noise (§5.3). A **calibration +tool** turns them into data-tuned, *interpretable* values: + +- **Fit the coefficients.** Run timed solves across a benchmark spread (varying + continuous / integer var counts, scenarios-per-bundle, nonant counts) and fit + `cont_coeff`, `int_weight`, `int_exponent`, `int_nonant_coeff` so that modeled + `effort(...)` tracks measured solve time. +- **Translate effort ↔ seconds.** Produce an effort→time scale so the abstract + budgets (`ef_effort_budget`, the base `M`, …) can be set and read in + **seconds** — the unit the `plus_*` budgets already use (minutes = seconds/60 + for humans) — e.g. "run the EF only if it should finish in ~600 s" instead of + opaque effort units. +- **Output.** Updated coefficients + the effort→seconds scale for a new **dated** + policy file — the producer side of the §5 "dated data files refined by + benchmark data" migration path. + +Caveats: solve time is machine- and solver-dependent and (for MIPs) noisy, so a +calibrated scale is per reference machine/solver and approximate. The **plus** +tier effectively *re-calibrates per session* via its measured single-scenario +time (`t₁`); **base** relies on this offline calibration. + +**Status: PR1.** Reviewers can't assess a policy whose effort coefficients and +budgets are *wild guesses*, so the calibration tool ships in PR1 and **the v1 +policy's numbers are its output** — an initial calibration on the **example set** +(the same timed solves the validator's run-tier performs, so they share +infrastructure), reproducible by re-running it. Preliminary but principled, and +far better than hand-guesses; a broader benchmark corpus (more models / machines) +is the future refinement that later dated files fold in. diff --git a/doc/src/generic_cylinders.rst b/doc/src/generic_cylinders.rst index ba189d077..ed6515ba9 100644 --- a/doc/src/generic_cylinders.rst +++ b/doc/src/generic_cylinders.rst @@ -8,6 +8,12 @@ run mpi-sppy. It provides command-line access to the hub-and-spoke system, the extensive form solver, confidence intervals, and many other features without requiring you to write a driver program. +.. tip:: + If you are new to mpi-sppy, add ``--out-of-the-box`` and let the driver + pick a sensible configuration automatically, then read the equivalent + command line it prints to learn the explicit options. See + :ref:`out_of_the_box`. + Your Model File (Module) ------------------------ diff --git a/doc/src/index.rst b/doc/src/index.rst index 71c846cb3..26a6e2843 100644 --- a/doc/src/index.rst +++ b/doc/src/index.rst @@ -25,6 +25,7 @@ MPI is used. :caption: Running with generic_cylinders generic_cylinders.rst + out_of_the_box.rst examples.rst ef.rst chance_constraints.rst diff --git a/doc/src/out_of_the_box.rst b/doc/src/out_of_the_box.rst new file mode 100644 index 000000000..1589b269c --- /dev/null +++ b/doc/src/out_of_the_box.rst @@ -0,0 +1,254 @@ +.. _out_of_the_box: + +Out-of-the-box auto-configuration +================================= + +The ``--out-of-the-box`` option lets a relatively new user obtain a *sensible* +mpi-sppy run with almost no knowledge of the library's internals. You supply a +model module (and its scenario data); mpi-sppy introspects the environment and +the model and assembles a defensible configuration -- algorithm, solver, spokes, +flexible rank split, and proper bundling -- instead of requiring a hand-crafted +hub/spoke command line. + +The spirit is *"here is my model, go,"* followed by a clear explanation of what +was chosen and how to do better. + +.. note:: + Out-of-the-box (OOTB) only *fills gaps*. **Any option you set explicitly + always wins** -- OOTB never overrides it. So you can start from + ``--out-of-the-box`` and override individual choices as you learn. + +Basic usage +----------- + +Add ``--out-of-the-box`` to an otherwise minimal ``generic_cylinders`` command +line (your model still needs its scenario count -- ``--num-scens`` for two-stage +problems, ``--branching-factors`` for multistage): + +.. code-block:: bash + + # serial -> too few ranks for cylinders, so OOTB solves the EF + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box + + # 3+ ranks available; OOTB decomposes when the problem is big/hard enough + mpiexec -np 3 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 6 --out-of-the-box + +OOTB prints the configuration it chose, the **equivalent explicit command line** +(so you can reproduce, learn from, and tweak it), runs the model, and then prints +a prioritized **Suggestions** list. + +.. note:: + OOTB decomposes only when it expects the decomposition to pay off. The base + tier estimates how long the monolithic EF would take and, if that is within + budget, solves the EF even when several ranks are available -- because for a + small or fast-solving model the EF *is* the right call. On a fast machine with + a commercial solver, the bundled examples (farmer, sizes, aircond) are cheap + enough that ``--out-of-the-box`` chooses the EF for all of them. To exercise + the cylinder path on a small problem, either request a spoke (any + decomposition flag, e.g. ``--lagrangian``, forces a decomposition) or use the + ``--out-of-the-box-minus`` tier, which has no size estimate and decomposes + whenever there are more than a couple of scenarios: + + .. code-block:: bash + + mpiexec -np 3 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 6 --out-of-the-box-minus + +What OOTB decides +----------------- + +In order, OOTB chooses: + +#. **Solver.** The first installed solver in a preference order (persistent + commercial, then commercial, then a free QP-capable solver, then LP/MIP-only). + An LP/MIP-only solver (cbc, glpk) automatically adds + ``--linearize-proximal-terms`` because it cannot take the quadratic PH prox. + If you pass ``--solver-name`` it is used as-is (and carried over to + ``--EF-solver-name`` if OOTB ends up solving the EF). +#. **Extensive form vs. decomposition.** With fewer than three ranks there is no + useful cylinder configuration (hub + at least two spokes), so OOTB solves the + **EF**. Above the rank floor, OOTB still solves the EF when the whole problem + is small enough to expect a quick monolithic solve (see *Effort and the EF + gate* below). Otherwise it decomposes. If you explicitly request a + decomposition (any spoke or a non-default hub) and have enough ranks, OOTB + never substitutes the EF. +#. **Spokes (a small, widened core).** Starting from a minimal core of one outer + bound (``--lagrangian``) and one inner/incumbent spoke (``--xhatshuffle``), + OOTB adds further spokes only while every cylinder would still keep at least a + couple of ranks. The preference is to give a few cylinders width rather than + pile on many weak single-rank spokes. So six ranks become three cylinders, + widened -- not six single-rank cylinders. +#. **Flexible rank split.** Ranks are split across cylinders unevenly: cheaper + cylinders (the xhat family) get a smaller share via per-spoke + ``--*-rank-ratio`` flags. (This is a crude cold-start split; the right split + depends on relative subproblem solve cost.) +#. **Proper bundling.** When there are many scenarios, OOTB forms proper bundles, + choosing the largest ``--scenarios-per-bundle`` that divides the scenario + count, leaves at least as many bundles as ranks, and keeps a bundle's modeled + solve effort within budget. +#. **A few extra defaults**, each backed off if you addressed the same concern: + ``--default-rho 1`` and the ``--grad-rho`` rho setter, ``--rel-gap 0.01``, + ``--max-iterations 100``, and ``--dynamic-rho-primal-crit``. + +Transparency +------------ + +OOTB prints the choices and the equivalent command line up front, and a +**Suggestions** list after the run (so the suggestions can reflect how the run +went). For example, a serial farmer run reports:: + + [out-of-the-box] tier 'base', policy 2026-06-28 + - solver: gurobi_persistent (first available in preference order) + - --EF: only 1 ranks; decomposition needs >= 3 + - --EF-solver-name gurobi_persistent: EF solver (gurobi_persistent) + [out-of-the-box] equivalent command line: + mpiexec -np 1 python -m mpi4py -m mpisppy.generic_cylinders \ + --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi_persistent + ... + [out-of-the-box] Suggestions: + * Ran the monolithic EF because only 1 MPI rank(s) were available; with + >= 3 ranks OOTB would decompose (hub + bound spokes). + +The equivalent command line is anchored with the module and scenario +specification and lists every flag OOTB added, so you can paste it (dropping +``--out-of-the-box``) to reproduce or modify the run. + +Effort tiers (how deeply OOTB inspects the model) +------------------------------------------------- + +Three mutually-exclusive flags select how deeply OOTB looks at the model; they +share one interpreter and one policy file and differ only in how much they +inspect. Every decision uses the best fact available and degrades gracefully to +a suggestion when a fact is missing. + +.. list-table:: + :header-rows: 1 + :widths: 22 14 64 + + * - Flag + - Instantiates + - What it can decide + * - ``--out-of-the-box-minus`` + - nothing + - EF gate by scenario *count*; solver by availability. Cannot size proper + bundles (no model size information). + * - ``--out-of-the-box`` *(default)* + - one probe scenario + - Size-aware EF gate; integrality- and size-aware bundling. The recommended + tier. + * - ``--out-of-the-box-plus`` + - *(reserved)* + - Planned: instantiate all scenarios and do a brief timed solve for + solve-time / gap information. Currently behaves like the base tier. + +``--out-of-the-box-plus`` is **not** an autotuner: it is "out-of-the-box with +more information," making the same one-shot decisions as the base tier. (It is +reserved for a future release; today it is equivalent to ``--out-of-the-box``.) + +Effort and the EF gate +^^^^^^^^^^^^^^^^^^^^^^^ + +The base tier instantiates one scenario to read its size profile (continuous / +integer variable counts and nonant counts) and models solve *effort* from it. +The numbers in the policy file are **calibrated to roughly seconds**, so the EF +budget reads as a wall-clock target: OOTB solves the EF when the whole problem's +modeled effort is within ``ef_effort_budget`` (about that many seconds), and +sizes bundles so a bundle is at most a small multiple as hard as a single +scenario. Integer content scales superlinearly, so an integer-heavy model +decomposes at a far smaller scenario count than a continuous one. + +``--inspect-only`` (dry run) +---------------------------- + +``--inspect-only`` does the inspection, prints the configuration, the equivalent +command line, and config-time suggestions, then **stops before the production +run**. It is independent of OOTB (on its own it just verifies that one scenario +instantiates), but pairs naturally with it: + +.. code-block:: bash + + # plan the run, print the equivalent command line, do not solve + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box --inspect-only + +``--inspect-only`` takes an optional **assumed rank count** for HPC planning: +``--inspect-only 512`` plans as if 512 ranks were available -- so you can get the +recommended command line for a large job *from a login node, without launching +it*. Everything else (installed solvers, model size) still comes from the real +session; only the rank count is hypothetical. + +Policy files +------------ + +OOTB's choices are driven by a dated, declarative **policy file** under +``mpisppy/generic/ootb_policies/`` -- data, not code, interpreted by a thin +Python routine so every decision is explainable. A bare ``--out-of-the-box`` +uses the newest shipped default policy; ``--out-of-the-box PATH`` uses the policy +file at ``PATH`` (the optional value of the flag *is* the path). Policy files +with different *foci* may ship side by side, distinguished by filename; you +select one by passing its path. The run logs which policy and ``policy_version`` +it used. + +The policy holds the solver preference order, the EF budget, the spoke ladder and +rank ratios, the bundle-effort model, and the extra-option defaults. Its numbers +are produced by the calibration tool (below), not hand-guessed. + +.. _ootb_validator: + +Validating a policy file +------------------------ + +A policy file is checked by a validator that confirms it is well-formed and that +its recommendations make sense -- and, on demand, actually run: + +.. code-block:: bash + + # static schema + decision checks on the default policy + python -m mpisppy.generic.ootb_validate + + # also exercise the real example models (needs a solver to instantiate) + python -m mpisppy.generic.ootb_validate --examples + + # also actually run the recommended configs and flag problem cases + python -m mpisppy.generic.ootb_validate --run --json report.json + +The validator has three layers: static schema checks (every referenced flag is +real, keys/types are right); decision checks (the EF gate fires when it should, +forced decomposition wins, bundling is valid, no conflicting rho setters); and +run checks that execute the recommended configurations and **flag** two cases for +a human to review -- an EF that misses a 1% gap in ten minutes, and cylinders +that max out on iterations. It produces a human-readable and a machine-readable +(``--json``) report. The fast, solver-free layers gate continuous integration; +the run tier is for nightly / local use. + +.. _ootb_calibrator: + +Calibrating the effort numbers +------------------------------ + +The effort coefficients and budget are produced by a calibration tool from timed +solves on the example models, so they track measured wall-clock time on a +reference machine rather than being guesses: + +.. code-block:: bash + + python -m mpisppy.generic.ootb_calibrate --solver-name gurobi \ + --output mpisppy/generic/ootb_policies/ootb_policy_.json + +The tool times extensive-form solves over a spread of bundle sizes, fits the +continuous / integer / nonant coefficients (choosing the integer exponent by best +fit), keeps the coefficients in seconds units so the budgets read as seconds, and +writes a new dated policy with provenance. Solve time is machine- and +solver-dependent (and noisy for MIPs), so a calibrated policy is per reference +machine and approximate; re-run the tool to recalibrate for your environment. + +Limitations +----------- + +OOTB aims for a *defensible* configuration, not an *optimal* one. It does not +tune convergence parameters to optimality and does not search a parameter space. +For full control, write an explicit hub/spoke command line (the rest of +:ref:`generic_cylinders` documents every option) -- and remember that OOTB emits +exactly such a command line for you to start from. diff --git a/doc/src/quick_start.rst b/doc/src/quick_start.rst index 76b6eea09..b2b31e666 100644 --- a/doc/src/quick_start.rst +++ b/doc/src/quick_start.rst @@ -377,6 +377,24 @@ guidance and HPC-specific tips, see :ref:`Install mpi4py`. Running the Farmer Example --------------------------- +**Recommended first run: let mpi-sppy configure itself.** Add +``--out-of-the-box`` and the driver introspects the environment and the model and +picks a sensible configuration (solver, EF vs. decomposition, spokes, bundling), +prints the equivalent explicit command line, and runs it: + +.. code-block:: bash + + python -m mpisppy.generic_cylinders --module-name farmer --num-scens 3 \ + --out-of-the-box + +Any option you set explicitly always wins, so this is a good starting point you +can refine. (For a small, fast-solving model like farmer, OOTB will sensibly +choose the extensive form; it decomposes for larger or harder problems.) See +:ref:`out_of_the_box` for the full description -- effort tiers, +``--inspect-only``, policy files, and the validation and calibration tools. + +The explicit forms below show what such a run is equivalent to. + **Solve the EF** (does not use MPI): .. code-block:: bash diff --git a/mpisppy/generic/ootb_calibrate.py b/mpisppy/generic/ootb_calibrate.py new file mode 100644 index 000000000..2085fc06b --- /dev/null +++ b/mpisppy/generic/ootb_calibrate.py @@ -0,0 +1,336 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) effort-calibration tool. + +The effort_scaling coefficients and the EF effort budget are abstract by +construction (arbitrary "effort units"); left as hand-guesses they are +unassessable. This tool turns them into data-tuned, interpretable values from +timed solves on the mpi-sppy examples (the producer side of the dated-policy +migration path -- design doc sec. 9). It: + + 1. FITS the effort_scaling shape -- cont_coeff, int_weight, int_exponent, + int_nonant_coeff -- so that modeled effort tracks measured solve time, by + timing extensive-form solves over a spread of bundle sizes across models of + differing continuous / integer / nonant content. + 2. CALIBRATES effort to ~seconds: the fitted coefficients are kept in seconds + units, so modeled effort approximates predicted solve time and the absolute + budgets read as roughly seconds (ef_effort_budget is set from the policy's + ef_target_seconds) instead of opaque large numbers. + 3. WRITES a new dated policy file with the fitted numbers in place of the + cold-start guesses. + + python -m mpisppy.generic.ootb_calibrate --solver-name gurobi \\ + --output mpisppy/generic/ootb_policies/ootb_policy_2026-07-01.json + +Caveats: solve time is machine- and solver-dependent and (for MIPs) noisy, so a +calibrated scale is per reference machine/solver and approximate. The fit is the +LINEAR part (cont/int/nonant coefficients) for each candidate integer exponent; +the best exponent is chosen by R^2. The pure fit (fit_effort_model) is +solver-free and unit-tested; the measurement needs a solver and is not run in CI. +""" + +from __future__ import annotations + +import argparse +import datetime +import json +import sys +import time + +import mpisppy.utils.sputils as sputils +from mpisppy.generic import out_of_the_box as ootb +from mpisppy.generic import ootb_validate as val + + +# Integer-exponent grid searched during the fit (int_exponent in the effort +# model). 1.0 = integers scale like LP; > 1 captures branch-and-bound blow-up. +EXPONENT_GRID = [1.0, 1.25, 1.5, 1.75, 2.0, 2.5, 3.0] + +# Bundle sizes (scenarios per EF solve) used to probe how solve time scales. +DEFAULT_SPB_GRID = [1, 2, 4, 8, 16] + +# How many times to solve each point (the minimum is kept -- a denoiser). +DEFAULT_REPS = 2 + + +# --------------------------------------------------------------------------- +# Fitting [pure; unit-tested] +# --------------------------------------------------------------------------- + + +def _round_sig(x: float, n: int = 6) -> float: + """Round to n significant figures. Decimal-place rounding would destroy the + legitimately tiny coefficients that multiply huge (int*spb)^exponent terms + (e.g. int_weight ~ 1e-9 when the exponent is 3 and there are ~150 integers).""" + import math + if x == 0 or not math.isfinite(x): + return 0.0 + return round(x, -int(math.floor(math.log10(abs(x)))) + (n - 1)) + + +def _design_columns(point: dict, exponent: float): + """The three effort features for one measured point, matching ootb._effort: + continuous content (~spb), integer content (^exponent), integer nonants + (fixed per bundle).""" + cont = point["vars_cont"] * point["spb"] + nint = point["vars_int"] * point["spb"] + return [cont, nint ** exponent, point["nonants_int"]] + + +def fit_effort_model(points: list, exponents=EXPONENT_GRID) -> dict: + """Fit the effort_scaling coefficients + an effort->seconds scale to timed + solves. + + points: list of dicts with PER-SCENARIO `vars_cont`, `vars_int`, + `nonants_int`, the bundle size `spb`, and the measured `seconds`. + + For each candidate integer exponent we solve a non-negative least squares for + (a, b, c) so that a*cont + b*int^p + c*nonant ~= seconds, and keep the + exponent with the best R^2. The fitted coefficients are kept in SECONDS units + (we do NOT divide the scale back out), so modeled effort approximates the + predicted solve time directly -- the absolute budgets (ef_effort_budget) then + read as roughly seconds (e.g. ~120) instead of opaque large numbers. + seconds_per_effort_unit is therefore ~1; it stays in the schema to document + the units and to let a focus rescale. Returns the effort_scaling fields plus + r2 / n_points. + """ + import numpy as np + from scipy.optimize import nnls + + if len(points) < 3: + raise ValueError(f"need at least 3 measured points to fit, got {len(points)}") + + y = np.array([p["seconds"] for p in points], dtype=float) + best = None + for p in exponents: + A = np.array([_design_columns(pt, p) for pt in points], dtype=float) + coef, _ = nnls(A, y) + ss_res = float(((A @ coef - y) ** 2).sum()) + ss_tot = float(((y - y.mean()) ** 2).sum()) + r2 = 1.0 - ss_res / ss_tot if ss_tot > 0 else 0.0 + if best is None or r2 > best["r2"]: + best = {"coef": coef, "p": p, "r2": r2} + + a, b, c = (float(x) for x in best["coef"]) + return { + "cont_coeff": _round_sig(a), + "int_weight": _round_sig(b), + "int_exponent": best["p"], + "int_nonant_coeff": _round_sig(c), + "seconds_per_effort_unit": 1.0, # effort is calibrated to ~seconds + "r2": round(best["r2"], 4), + "n_points": len(points), + } + + +# --------------------------------------------------------------------------- +# Measurement [needs a solver; not run in CI] +# --------------------------------------------------------------------------- + + +def _pick_solver(policy: dict, requested: str | None) -> str: # pragma: no cover + """A non-persistent solver for one-shot EF solves (persistent interfaces + need set_instance and add setup overhead that would pollute the timing).""" + if requested: + return requested + for name in policy["solver"]["preference_order"]: + plain = name.replace("_persistent", "") + if plain in ootb._detect_available_solvers([plain]): + return plain + raise RuntimeError("no known solver available; pass --solver-name") + + +def _time_ef_solve(names, scenario_creator, kwargs, solver_name, reps) -> float: # pragma: no cover + """Build an EF over `names` scenarios and return the MIN wall-clock solve + time over `reps` solves (model build excluded from the timing).""" + import pyomo.environ as pyo + best = None + for _ in range(reps): + ef = sputils.create_EF(names, scenario_creator, kwargs, + suppress_warnings=True) + solver = pyo.SolverFactory(solver_name) + t0 = time.perf_counter() + solver.solve(ef) + dt = time.perf_counter() - t0 + best = dt if best is None else min(best, dt) + return best + + +def measure_example(spec: dict, solver_name: str, spb_grid, reps) -> list: # pragma: no cover + """Probe one example's per-scenario size profile, then time EF solves over a + range of bundle sizes. Returns a list of measured points.""" + module = val._load_example_module(spec) + cfg = val._example_cfg(spec, module) + kwargs = module.kw_creator(cfg) + profile = ootb._size_profile(ootb._build_probe_scenario(module, cfg)) + total = ootb._detect_num_scens(module, cfg) + points = [] + for spb in spb_grid: + if spb > total: + break + names = module.scenario_names_creator(spb) + try: + seconds = _time_ef_solve(names, module.scenario_creator, kwargs, + solver_name, reps) + except Exception as e: # noqa: BLE001 + print(f" [{spec['name']} spb={spb}] solve failed: " + f"{type(e).__name__}: {e}", file=sys.stderr) + continue + pt = {"example": spec["name"], "spb": spb, + "vars_cont": profile["vars_cont"], "vars_int": profile["vars_int"], + "nonants_int": profile["nonants_int"], "seconds": round(seconds, 6)} + points.append(pt) + print(f" {spec['name']:8} spb={spb:<3} " + f"cont={profile['vars_cont']} int={profile['vars_int']} " + f"nonants_int={profile['nonants_int']} -> {seconds:.4f}s") + return points + + +def _calibration_specs() -> list: + """Example specs with scenario counts tuned for calibration: enough + scenarios that several bundle sizes are reachable (the validator's run-tier + deliberately uses tiny counts; calibration wants a wider spread, and more + integer points come from a larger MIP EF).""" + counts = {"farmer": {"num_scens": 16}, "sizes": {"num_scens": 10}, + "aircond": {"branching_factors": [4, 4]}} + specs = [] + for spec in val.example_models(): + s = dict(spec) + s["scens"] = counts.get(spec["name"], spec["scens"]) + specs.append(s) + return specs + + +def collect_points(policy: dict, solver_name: str, spb_grid, reps, + specs=None) -> list: # pragma: no cover + points = [] + for spec in (specs if specs is not None else _calibration_specs()): + print(f"[calibrate] timing {spec['name']} ({spec['kind']}) ...") + points.extend(measure_example(spec, solver_name, spb_grid, reps)) + return points + + +# --------------------------------------------------------------------------- +# Apply the fit to a policy +# --------------------------------------------------------------------------- + + +def calibrated_policy(base_policy: dict, fit: dict, points: list, + solver_name: str, today: str) -> dict: + """Return a copy of base_policy with the fitted effort_scaling, the + effort->seconds scale, and a seconds-derived ef_effort_budget.""" + import copy + pol = copy.deepcopy(base_policy) + + es = pol["effort_scaling"] + es["cont_coeff"] = fit["cont_coeff"] + es["int_weight"] = fit["int_weight"] + es["int_exponent"] = fit["int_exponent"] + es["int_nonant_coeff"] = fit["int_nonant_coeff"] + es["seconds_per_effort_unit"] = round(fit["seconds_per_effort_unit"], 8) + es["_calibration"] = { + "solver": solver_name, "r2": fit["r2"], "n_points": fit["n_points"], + "date": today, "note": "Fitted by mpisppy.generic.ootb_calibrate on the " + "example set; per reference machine/solver and approximate. " + "seconds_per_effort_unit: modeled effort * this ~= seconds."} + es.pop("_cold_start_guess", None) # these numbers are now data-tuned + # Reconcile the prose: the coefficients are no longer cold-start guesses. + es["_comment"] = es.get("_comment", "").replace( + "All cold-start guesses; foci ship different shapes; benchmark data refines.", + "Coefficients are calibrated by ootb_calibrate (see _calibration); foci " + "may ship different shapes; more benchmark data refines further.") + + # Derive the EF budget in effort units from the seconds target so the budget + # is consistent with the fitted scale (effort <= seconds / sec_per_effort). + ef = pol["ef_fallback"] + spe = fit["seconds_per_effort_unit"] + if spe > 0 and ef.get("ef_target_seconds"): + ef["ef_effort_budget"] = int(round(ef["ef_target_seconds"] / spe)) + ef["_calibration_note"] = ("ef_effort_budget derived from " + "ef_target_seconds / seconds_per_effort_unit " + f"(calibrated {today}).") + guesses = ef.get("_cold_start_guess", []) + ef["_cold_start_guess"] = [g for g in guesses if g != "ef_effort_budget"] + ef["_comment"] = ef.get("_comment", "").replace( + "All cold-start guesses.", + "ef_effort_budget is calibrated (see _calibration_note); " + "ef_target_seconds and ef_if_num_scens_at_most remain authored guesses.") + + pol["policy_version"] = today + pol["provenance"] = (f"CALIBRATED {today} by mpisppy.generic.ootb_calibrate " + f"(solver {solver_name}, R^2={fit['r2']}, " + f"{fit['n_points']} timed EF solves on the example set). " + "effort_scaling and ef_effort_budget are data-tuned; " + "remaining numbers are still authored. Per reference " + "machine/solver and approximate (MIP times are noisy).") + return pol + + +# --------------------------------------------------------------------------- +# Orchestration + CLI +# --------------------------------------------------------------------------- + + +def run_calibration(base_policy_path, solver_name=None, spb_grid=DEFAULT_SPB_GRID, + reps=DEFAULT_REPS, today=None): # pragma: no cover + """Measure, fit, and return (calibrated_policy_dict, fit, points).""" + base = ootb.load_policy(base_policy_path or None) + solver = _pick_solver(base, solver_name) + print(f"[calibrate] solver: {solver}") + points = collect_points(base, solver, spb_grid, reps) + fit = fit_effort_model(points) + if today is None: + today = datetime.date.today().isoformat() + pol = calibrated_policy(base, fit, points, solver, today) + return pol, fit, points + + +def main(argv=None): # pragma: no cover + p = argparse.ArgumentParser( + prog="python -m mpisppy.generic.ootb_calibrate", + description="Calibrate OOTB effort_scaling from timed example solves " + "(design doc sec. 9).") + p.add_argument("--base", default="", + help="base policy to start from (default: shipped default)") + p.add_argument("--solver-name", default=None, + help="solver for the timed solves (default: first available)") + p.add_argument("--output", default=None, + help="write the calibrated policy here (default: print only)") + p.add_argument("--reps", type=int, default=DEFAULT_REPS, + help=f"solves per point, min kept (default {DEFAULT_REPS})") + p.add_argument("--spb", default=None, + help="comma-separated bundle sizes (default " + f"{','.join(map(str, DEFAULT_SPB_GRID))})") + args = p.parse_args(argv) + + spb_grid = ([int(x) for x in args.spb.split(",")] if args.spb + else DEFAULT_SPB_GRID) + pol, fit, points = run_calibration(args.base, args.solver_name, spb_grid, + args.reps) + + print("\n[calibrate] fit:") + for k in ("cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff", + "seconds_per_effort_unit", "r2", "n_points"): + print(f" {k}: {fit[k]}") + print(f" ef_effort_budget -> {pol['ef_fallback']['ef_effort_budget']} " + f"(from {pol['ef_fallback'].get('ef_target_seconds')}s target)") + + if args.output: + with open(args.output, "w") as fp: + json.dump(pol, fp, indent=2) + print(f"\n[calibrate] wrote calibrated policy to {args.output}") + print("[calibrate] validate it with: python -m mpisppy.generic." + f"ootb_validate {args.output}") + else: + print("\n[calibrate] (no --output given; not writing a policy file)") + return 0 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(main()) diff --git a/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json new file mode 100644 index 000000000..0374612dd --- /dev/null +++ b/mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json @@ -0,0 +1,245 @@ +{ + "_description": "Out-of-the-box (OOTB) auto-configuration policy. This is DATA, not code: a thin Python interpreter reads it and a set of facts about the environment (MPI ranks, installed solvers, optional cores/memory) and the model (scenario count, stage structure) to assemble a defensible mpi-sppy run. User-supplied options always win; OOTB only fills gaps. See doc/designs/out_of_the_box_design.md.", + "schema_version": 1, + "policy_version": "2026-06-28", + "provenance": "CALIBRATED 2026-06-28 by mpisppy.generic.ootb_calibrate (solver gurobi, R^2=0.9995, 14 timed EF solves on the example set). effort_scaling and ef_effort_budget are data-tuned; remaining numbers are still authored. Per reference machine/solver and approximate (MIP times are noisy).", + "ef_fallback": { + "_comment": "When to solve the monolithic EF instead of decomposing. Always EF when ranks < min_ranks_for_decomposition (requirement 3). Otherwise EF when the WHOLE problem is small enough, measured with the SAME effort model as bundle sizing (effort_scaling) applied to all scenarios as one model, i.e. effort(num_scens). Unlike bundle sizing's RELATIVE budget, the EF budget is ABSOLUTE (the monolith has no single-scenario reference). base: effort(num_scens) <= ef_effort_budget (same effort units as bundle effort). plus: measured_single_scenario_seconds * effort(num_scens)/effort(1) <= ef_target_seconds. minus (no size profile): fall back to the count rule num_scens <= ef_if_num_scens_at_most. ef_effort_budget is calibrated (see _calibration_note); ef_target_seconds and ef_if_num_scens_at_most remain authored guesses.", + "min_ranks_for_decomposition": 3, + "ef_if_num_scens_at_most": 2, + "ef_effort_budget": 120, + "ef_target_seconds": 120, + "_cold_start_guess": [ + "ef_if_num_scens_at_most", + "ef_target_seconds" + ], + "_calibration_note": "ef_effort_budget derived from ef_target_seconds / seconds_per_effort_unit (calibrated 2026-06-28)." + }, + "solver": { + "_comment": "The base/plus probe classifies the model into one of six problem classes (LP, MIP, QP, MIQP, NLP, MINLP) from its integrality (vars_int) and the max polynomial degree of its objective/constraints (linear / quadratic / more-than-quadratic). OOTB then picks the first AVAILABLE solver from that class's list in preference_order_by_class -- e.g. a more-than-quadratic (NLP) model routes to ipopt, an integer model never routes to ipopt. Availability is SolverFactory(name).available(exception_flag=False). Persistence is requested by NAMING the *_persistent interface (no separate flag), so persistent commercial comes first within each class. preference_order is the MASTER superset (the detection candidate set, and the fallback when the class is unknown -- the minus tier does not instantiate, so it cannot classify). Every per-class list is a subset of it (the validator enforces this). Continuous QP/NLP lists drop the LP/MIP-only solvers (cbc/glpk); integer lists drop ipopt (continuous only); MINLP lists dedicated global solvers, which are rarely installed (OOTB then warns).", + "preference_order": [ + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent", + "gurobi", + "cplex", + "xpress", + "appsi_highs", + "highs", + "ipopt", + "cbc", + "glpk", + "baron", + "scip", + "bonmin", + "couenne" + ], + "preference_order_by_class": { + "_comment": "Per-problem-class preferred solver lists (design sec. 6). Keyed by class: LP, MIP, QP, MIQP, NLP, MINLP. Each list is tried top-to-bottom; the first installed solver wins. Every entry must also appear in preference_order. Note: on the DECOMPOSITION path PH adds a quadratic proximal term, so an LP/MIP model's subproblem is really a QP/MIQP; when the chosen solver is LP/MIP-only (lp_mip_only_force_linearize_prox) OOTB linearizes that prox instead of requiring a QP-capable solver, which is why cbc/glpk remain acceptable for LP/MIP.", + "LP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "cbc", "glpk"], + "MIP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "cbc", "glpk"], + "QP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs", "ipopt"], + "MIQP": ["gurobi_persistent", "cplex_persistent", "xpress_persistent", "gurobi", "cplex", "xpress", "appsi_highs", "highs"], + "NLP": ["ipopt"], + "MINLP": ["baron", "scip", "bonmin", "couenne"] + }, + "commercial": [ + "gurobi", + "cplex", + "xpress", + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent" + ], + "qp_capable": [ + "gurobi", + "cplex", + "xpress", + "gurobi_persistent", + "cplex_persistent", + "xpress_persistent", + "ipopt", + "highs", + "appsi_highs" + ], + "lp_mip_only_force_linearize_prox": [ + "glpk", + "cbc" + ], + "caveats": { + "_comment": "Human-readable notes about individual solvers. The base/plus probe now DOES introspect integrality and nonlinearity to route solver selection (see preference_order_by_class), so these caveats mostly explain the routing; the remaining ones are surfaced via suggestions.", + "ipopt": "Continuous NLP only; cannot handle integer variables (so it is in the NLP/QP lists, never the MIP/MIQP/MINLP lists).", + "highs": "QP ok; mixed-integer quadratic (MIQP) support is limited, so the PH prox on MIP subproblems may be rejected.", + "glpk_cbc": "LP/MIP only; cannot handle the quadratic PH prox term, so OOTB sets --linearize-proximal-terms." + } + }, + "hub": { + "_comment": "PH is the default hub in the generic driver; no flag is emitted to select it.", + "default_factory": "ph_hub", + "flag": null + }, + "spoke_ladder": { + "_comment": "Ordered list of spokes to switch on as ranks allow, lowest priority number first. Only flags WIRED in mpisppy/generic/spokes.py appear here. The 3-rank minimum fills priorities 1 and 2 (one outer + one inner bound) so there is always an optimality gap. Each rung consumes one cylinder; see rank_allocation for how leftover ranks are spent. NOTE: the ORDERING of priorities 3-6 is itself a cold-start guess (no benchmark data yet); _cold_start_guess lists only real keys, so it cannot name the ordering, but treat the priority order below as provisional.", + "rungs": [ + { + "priority": 1, + "flag": "--lagrangian", + "factory": "lagrangian_spoke", + "bound": "outer", + "note": "Lagrangian dual bound; cheap and robust. Core." + }, + { + "priority": 2, + "flag": "--xhatshuffle", + "factory": "xhatshuffle_spoke", + "bound": "inner", + "note": "Incumbent finder over shuffled scenarios. Core." + }, + { + "priority": 3, + "flag": "--fwph", + "factory": "fwph_spoke", + "bound": "outer", + "note": "Frank-Wolfe PH; often tightens the outer bound." + }, + { + "priority": 4, + "flag": "--xhatxbar", + "factory": "xhatxbar_spoke", + "bound": "inner", + "note": "Cheap second incumbent source from xbar." + }, + { + "priority": 5, + "flag": "--subgradient", + "factory": "subgradient_spoke", + "bound": "outer", + "note": "Alternative dual bound." + }, + { + "priority": 6, + "flag": "--reduced-costs", + "factory": "reduced_costs_spoke", + "bound": "outer", + "note": "More advanced; benefits markedly from a persistent solver. Last." + } + ], + "core_roster_min": { + "_comment": "Minimum spokes for a meaningful run: at least one of each bound type. With the PH hub this is the 3-rank floor.", + "outer": 1, + "inner": 1 + }, + "max_cylinders": 7, + "_cold_start_guess": [ + "max_cylinders" + ] + }, + "rank_allocation": { + "_comment": "Two parts. (A) ROSTER SIZE: prefer a SMALL CORE (>=1 outer + >=1 inner spoke) widened by ranks over a wide roster of weak spokes. Start from the core; add further ladder rungs only while each cylinder would still get >= min_ranks_per_cylinder ranks (uniform/nominal basis -- a coarse gate) and we stay <= max_cylinders. So 6 ranks -> hub + lagrangian + xhatshuffle (3 cylinders, widened), NOT 6 single-rank cylinders. (B) UNBALANCED DISTRIBUTION (flex-ranks): split ranks across the chosen cylinders by per-cylinder RATIOS, not uniformly -- cheaper cylinders get fewer ranks. ratios are normalized over the chosen cylinders and floored at 1 rank each. WARNING: rank_ratios here are a CRUDE cold-start (e.g. xhatter 0.2); the right split is a much more complicated calculation that depends on the NATURE OF THE SUBPROBLEMS (relative solve cost), ideally informed by the plus tier's measurements.", + "min_ranks_per_cylinder": 2, + "default_rank_ratio": 1.0, + "rank_ratios": { + "--xhatshuffle": 0.2, + "--xhatxbar": 0.2, + "--xhatlshaped": 0.2 + }, + "_cold_start_guess": [ + "min_ranks_per_cylinder", + "default_rank_ratio", + "rank_ratios" + ] + }, + "effort_scaling": { + "_comment": "SHAPE of how solve effort grows with the size of a (sub)problem. Shared by bundle sizing at every tier (design doc Bundle sizing section). effort(n) for a bundle of n scenarios = cont_coeff*(n*vars_cont) + int_weight*(n*vars_int)^int_exponent + int_nonant_coeff*nonants_int, using the probe size profile (vars_cont, vars_int, nonants_int). The base tier uses this RELATIVE (ratio vs a single scenario); the plus tier uses the SAME shape CALIBRATED by a measured single-scenario solve time. int_exponent>1 captures MIP superlinearity; =1 treats integers like LP. Coefficients are calibrated by ootb_calibrate (see _calibration); foci may ship different shapes; more benchmark data refines further.", + "cont_coeff": 2.46332e-05, + "int_weight": 1.57234e-09, + "int_exponent": 3.0, + "int_nonant_coeff": 5.96467e-05, + "seconds_per_effort_unit": 1.0, + "_calibration": { + "solver": "gurobi", + "r2": 0.9995, + "n_points": 14, + "date": "2026-06-28", + "note": "Fitted by mpisppy.generic.ootb_calibrate on the example set; per reference machine/solver and approximate. seconds_per_effort_unit: modeled effort * this ~= seconds." + } + }, + "bundle_sizing": { + "_comment": "How big should proper bundles be (not just whether to bundle)? Proper bundling only (loose bundling removed in 2026); --scenarios-per-bundle MUST divide num_scens; library requires number_of_bundles >= number_of_ranks. The MINUS tier CANNOT bundle: with no size profile there is no safe way to size a bundle, so minus always runs unbundled (bundling needs base or plus). For base/plus: pick the LARGEST scenarios_per_bundle that (a) divides num_scens, (b) leaves >= max(intra_ranks, min_bundles_per_intra_rank*intra_ranks) bundles, and (c) keeps a bundle's modeled effort within budget. base budget is RELATIVE: effort(spb)/effort(1) <= base_max_hardness_vs_single_scenario (unitless, portable, no measurement). plus budget is ABSOLUTE: measured_single_scenario_seconds * effort(spb)/effort(1) <= plus_target_seconds_per_bundle, with the measuring solve capped at plus_probe_solve_time_cap_seconds. Skip bundling below min_scens_to_consider_bundling. MIP solve times are noisy/non-monotone, so the shape in effort_scaling acts as a prior; a single measurement only sets the scale.", + "min_scens_to_consider_bundling": 50, + "min_bundles_per_intra_rank": 1, + "never_fewer_bundles_than_ranks": true, + "base_max_hardness_vs_single_scenario": 10, + "plus_target_seconds_per_bundle": 30, + "plus_probe_solve_time_cap_seconds": 60, + "_cold_start_guess": [ + "min_scens_to_consider_bundling", + "min_bundles_per_intra_rank", + "base_max_hardness_vs_single_scenario", + "plus_target_seconds_per_bundle", + "plus_probe_solve_time_cap_seconds" + ] + }, + "option_categories": { + "_comment": "Named option blocks OOTB applies by default, one per CONCERN, beyond the structural choices (solver / EF / spokes / bundling). Each is applied UNLESS the user already set any flag in its 'superseded_by' list -- the robust, per-concern form of 'user options win' (the user may address a concern with a DIFFERENT flag than OOTB's default). 'value' null = a boolean flag. These are decomposition-run options (skipped on the EF path). Foci may change them. All cold-start.", + "base_rho": { + "flag": "--default-rho", + "value": "1", + "superseded_by": [ + "--default-rho" + ], + "_note": "PH needs a base rho value. The grad-rho setter REFINES rho but does not itself supply a base, and (unlike sep/coeff/sensi rho) mpi-sppy does not auto-default it, so a decomposition with --grad-rho and no --default-rho is a hard error. Set a sane base here; the rho setter adapts from it." + }, + "rho_setter": { + "flag": "--grad-rho", + "value": null, + "superseded_by": [ + "--grad-rho", + "--sensi-rho", + "--coeff-rho", + "--sep-rho", + "--reduced-costs-rho" + ], + "_note": "mpi-sppy enforces that only ONE rho setter is active, so OOTB adding --grad-rho when the user already picked another rho setter would be a HARD ERROR -- this is exactly why superseded_by lists all of them, not just --grad-rho." + }, + "termination": { + "flag": "--rel-gap", + "value": "0.01", + "superseded_by": [ + "--rel-gap", + "--abs-gap" + ] + }, + "max_iterations": { + "flag": "--max-iterations", + "value": "100", + "superseded_by": [ + "--max-iterations" + ] + }, + "dynamic_rho": { + "flag": "--dynamic-rho-primal-crit", + "value": null, + "superseded_by": [ + "--dynamic-rho-primal-crit", + "--dynamic-rho-dual-crit" + ], + "_note": "Boolean flag (no argument needed); its paired threshold dynamic_rho_primal_thresh defaults to 0.1. Requires an active rho setter (config errors otherwise), which OOTB's default --grad-rho provides." + }, + "_cold_start_guess": [ + "base_rho", + "rho_setter", + "termination", + "max_iterations", + "dynamic_rho" + ] + }, + "additional_options": { + "_comment": "Catch-all for any OTHER extra flags not worth a named category. Each entry: flag, value (null = boolean), and optional superseded_by (defaults to [its own flag], i.e. plain per-flag override). For concern-level override (e.g. rho setters) use option_categories instead. Decomposition-run options (skipped on the EF path).", + "options": [] + }, + "suggestions": { + "_comment": "Requirement 4: a prioritized 'Suggestions' list, printed AFTER the run (so it can also reflect how the run went). The messages are MOSTLY COMPUTED in code (suggestion generators in mpisppy/generic/out_of_the_box.py): each inspects facts/decision/run-outcome and builds text with live values, so the PROSE lives in code, not here. Note the split from design sec. 5: decisions stay data-driven (this policy); suggestions are diagnostics and are computed. This block only TOGGLES/TUNES them -- 'disabled' lists generator names to suppress; thresholds (if any) can be added here for generators to read.", + "disabled": [] + } +} \ No newline at end of file diff --git a/mpisppy/generic/ootb_validate.py b/mpisppy/generic/ootb_validate.py new file mode 100644 index 000000000..91d660da4 --- /dev/null +++ b/mpisppy/generic/ootb_validate.py @@ -0,0 +1,730 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) policy-file validator. + +Given a dated policy file, check it is well-formed and that its recommended +configurations make sense -- and, on demand, that they actually RUN -- using the +mpi-sppy examples as test models. See doc/designs/out_of_the_box_design.md (sec. +8). It does NOT assert "expected results" (there is no cheap oracle); instead it +records what happened and FLAGS two run failure modes for a human to review. + + python -m mpisppy.generic.ootb_validate # layers 1+2 + python -m mpisppy.generic.ootb_validate --examples # +real models + python -m mpisppy.generic.ootb_validate --run # +layer 3 runs + +Three layers, fast to slow: + + 1. STATIC (schema): JSON parses; required keys/types; every referenced flag is + a real CLI option; spoke rungs / rho setters / DECOMPOSITION_FLAGS all line + up with the actual generic_cylinders vocabulary. + 2. DECISION (recommend() only, no solves): on hand-built synthetic Facts (the + CI-gating subset) and, with --examples, on probe-instantiated real models. + Asserts EF-when-it-should, decompose-when-it-should, forced-decomp-wins, + bundling validity, and no conflicting rho setters. + 3. RUN (--run; slow, needs a solver, NEVER a CI gate): actually run the + recommended configs on the small examples and FLAG (a) an EF that misses a + 1% gap in ten minutes and (b) cylinders that max out on iterations. + +The CI gate (mpisppy/tests/test_ootb_validate.py) runs only layers 1 + +2-synthetic on the shipped policy file(s) -- solver-free, fast. +""" + +from __future__ import annotations + +import argparse +import importlib +import json +import os +import re +import subprocess +import sys +import time +from dataclasses import dataclass + +import mpisppy.utils.config as config +from mpisppy.generic import parsing +from mpisppy.generic import out_of_the_box as ootb + + +# Validator settings (NOT policy): the two layer-3 auto-flag thresholds. +EF_GAP_TARGET = 0.01 # flag an EF that misses this MIP gap ... +EF_TIME_LIMIT_SEC = 600 # ... within this wall-clock budget (ten minutes) + + +# --------------------------------------------------------------------------- +# Result carriers +# --------------------------------------------------------------------------- + + +@dataclass +class Check: + layer: str # "static" | "decision" + name: str + ok: bool + detail: str = "" + + def as_dict(self) -> dict: + return {"layer": self.layer, "name": self.name, "ok": self.ok, + "detail": self.detail} + + +@dataclass +class RunRecord: + example: str + env: dict + mode: str # "EF" | "decompose" + returncode: int | None = None + walltime: float | None = None + objective: float | None = None + rel_gap: float | None = None + iterations: int | None = None + flagged: bool = False + flag_reason: str = "" + detail: str = "" + + def as_dict(self) -> dict: + return {k: getattr(self, k) for k in + ("example", "env", "mode", "returncode", "walltime", "objective", + "rel_gap", "iterations", "flagged", "flag_reason", "detail")} + + +# --------------------------------------------------------------------------- +# The example models the validator exercises (layers 2-examples and 3). +# --------------------------------------------------------------------------- + + +def _repo_root() -> str: + # mpisppy/generic/ootb_validate.py -> repo root is three directories up. + return os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + + +def example_models() -> list[dict]: + root = _repo_root() + return [ + {"name": "farmer", "kind": "2-stage continuous", + "dir": os.path.join(root, "examples", "farmer"), "module": "farmer", + "scens": {"num_scens": 6}}, + {"name": "sizes", "kind": "2-stage MIP", + "dir": os.path.join(root, "examples", "sizes"), "module": "sizes", + "scens": {"num_scens": 3}}, + {"name": "aircond", "kind": "multistage", + "dir": os.path.join(root, "mpisppy", "tests", "examples"), + "module": "aircond", + "scens": {"branching_factors": [3, 2]}}, + ] + + +def _scen_cli(spec: dict) -> list: + """The --num-scens / --branching-factors CLI args for a spec. Branching + factors are passed as ONE space-joined token (the ListOf(int) domain parses + a single string, so '--branching-factors 3 2' would leave '2' unparsed).""" + s = spec["scens"] + if "num_scens" in s: + return ["--num-scens", str(s["num_scens"])] + return ["--branching-factors", " ".join(str(b) for b in s["branching_factors"])] + + +def _child_env() -> dict: + """Environment for subprocess runs with this (singleton-MPI) process's MPI + variables scrubbed. Importing mpi-sppy initializes mpi4py.MPI here, so the + parent IS an MPI process; leaving OMPI_/PMI_/... in the child's environment + makes a fresh mpiexec think it is being relaunched inside a rank and fail.""" + return {k: v for k, v in os.environ.items() + if not k.startswith(("OMPI_", "PMI_", "PMIX_", "MPIR_", "HYDRA_"))} + + +# --------------------------------------------------------------------------- +# Shared helpers +# --------------------------------------------------------------------------- + + +def valid_flags() -> set: + """The authoritative set of real CLI flags, from the same declaration the + driver uses (parsing.add_driver_args with no model module).""" + ref = config.Config() + parsing.add_driver_args(ref) + return {"--" + name.replace("_", "-") for name in ref} + + +def _is_number(x) -> bool: + return isinstance(x, (int, float)) and not isinstance(x, bool) + + +# --------------------------------------------------------------------------- +# Layer 1: static (schema) checks +# --------------------------------------------------------------------------- + + +def validate_static(policy: dict) -> list: + checks = [] + flags = valid_flags() + + def add(name, ok, detail=""): + checks.append(Check("static", name, ok, detail)) + + # required top-level keys + required = ["schema_version", "policy_version", "ef_fallback", "solver", + "hub", "spoke_ladder", "rank_allocation", "effort_scaling", + "bundle_sizing", "option_categories", "additional_options", + "suggestions"] + missing = [k for k in required if k not in policy] + add("required top-level keys present", not missing, + f"missing: {missing}" if missing else "all present") + if missing: + return checks # the rest assume the structure exists + + # ef_fallback numbers + ef = policy["ef_fallback"] + add("ef_fallback.min_ranks_for_decomposition is a positive int", + isinstance(ef.get("min_ranks_for_decomposition"), int) + and ef["min_ranks_for_decomposition"] >= 1, + str(ef.get("min_ranks_for_decomposition"))) + for k in ("ef_if_num_scens_at_most", "ef_effort_budget", "ef_target_seconds"): + add(f"ef_fallback.{k} is a positive number", + _is_number(ef.get(k)) and ef[k] > 0, str(ef.get(k))) + + # solver + sp = policy["solver"] + pref = sp.get("preference_order") + add("solver.preference_order is a non-empty list of strings", + isinstance(pref, list) and len(pref) > 0 + and all(isinstance(s, str) for s in pref)) + for key in ("commercial", "qp_capable", "lp_mip_only_force_linearize_prox"): + val = sp.get(key, []) + add(f"solver.{key} is a subset of preference_order", + isinstance(val, list) and set(val) <= set(pref or []), + f"stray: {sorted(set(val) - set(pref or []))}") + # per-class preferred solver lists: all six classes present, each a non-empty + # subset of the master preference_order (so detection always probes them). + by_class = {k: v for k, v in sp.get("preference_order_by_class", {}).items() + if not k.startswith("_")} + add("solver.preference_order_by_class covers all six problem classes", + set(by_class) == {"LP", "MIP", "QP", "MIQP", "NLP", "MINLP"}, + f"have {sorted(by_class)}") + for cls, lst in by_class.items(): + add(f"solver.preference_order_by_class.{cls} is a non-empty subset of " + f"preference_order", + isinstance(lst, list) and len(lst) > 0 + and all(isinstance(s, str) for s in lst) + and set(lst) <= set(pref or []), + f"stray: {sorted(set(lst) - set(pref or []))}") + + # spoke ladder + ladder = policy["spoke_ladder"] + rungs = ladder.get("rungs", []) + rung_flags = [r.get("flag") for r in rungs] + add("spoke_ladder.rungs flags are real CLI options", + all(f in flags for f in rung_flags), + f"unknown: {[f for f in rung_flags if f not in flags]}") + add("spoke_ladder.rungs flags are decomposition (spoke) flags", + all(f in ootb.DECOMPOSITION_FLAGS for f in rung_flags), + f"not spokes: {[f for f in rung_flags if f not in ootb.DECOMPOSITION_FLAGS]}") + add("spoke_ladder.rungs have unique priorities", + len({r.get("priority") for r in rungs}) == len(rungs)) + add("spoke_ladder.rungs bounds are outer/inner", + all(r.get("bound") in ("outer", "inner") for r in rungs)) + core = ladder.get("core_roster_min", {}) + add("spoke_ladder.core_roster_min has outer & inner counts", + isinstance(core.get("outer"), int) and isinstance(core.get("inner"), int)) + add("spoke_ladder.max_cylinders is an int >= core roster size", + isinstance(ladder.get("max_cylinders"), int) + and ladder["max_cylinders"] >= 1 + core.get("outer", 0) + core.get("inner", 0)) + # the ladder must be able to satisfy the core roster + avail = {"outer": 0, "inner": 0} + for r in rungs: + if r.get("bound") in avail: + avail[r["bound"]] += 1 + need = {k: core.get(k, 0) for k in ("outer", "inner")} + add("spoke_ladder can satisfy core_roster_min", + avail["outer"] >= need["outer"] and avail["inner"] >= need["inner"], + f"available {avail} vs needed {need}") + + # rank allocation + ra = policy["rank_allocation"] + add("rank_allocation.min_ranks_per_cylinder is a positive int", + isinstance(ra.get("min_ranks_per_cylinder"), int) + and ra["min_ranks_per_cylinder"] >= 1) + add("rank_allocation.default_rank_ratio is a positive number", + _is_number(ra.get("default_rank_ratio")) and ra["default_rank_ratio"] > 0) + rr = ra.get("rank_ratios", {}) + add("rank_allocation.rank_ratios keys are real spoke flags", + all(f in flags for f in rr), f"unknown: {[f for f in rr if f not in flags]}") + add("rank_allocation.rank_ratios values are positive numbers", + all(_is_number(v) and v > 0 for v in rr.values())) + + # effort scaling + es = policy["effort_scaling"] + for k in ("cont_coeff", "int_weight", "int_exponent", "int_nonant_coeff"): + add(f"effort_scaling.{k} is a non-negative number", + _is_number(es.get(k)) and es[k] >= 0, str(es.get(k))) + add("effort_scaling.int_exponent >= 1 (integers at least linear)", + _is_number(es.get("int_exponent")) and es["int_exponent"] >= 1) + + # bundle sizing + bs = policy["bundle_sizing"] + for k in ("min_scens_to_consider_bundling", "min_bundles_per_intra_rank", + "base_max_hardness_vs_single_scenario", + "plus_target_seconds_per_bundle", "plus_probe_solve_time_cap_seconds"): + add(f"bundle_sizing.{k} is a positive number", + _is_number(bs.get(k)) and bs[k] > 0, str(bs.get(k))) + add("bundle_sizing.base_max_hardness_vs_single_scenario >= 1", + _is_number(bs.get("base_max_hardness_vs_single_scenario")) + and bs["base_max_hardness_vs_single_scenario"] >= 1) + + # option categories + oc = policy.get("option_categories", {}) + for name, cat in oc.items(): + if name.startswith("_"): + continue + add(f"option_categories.{name}.flag is a real CLI option", + cat.get("flag") in flags, str(cat.get("flag"))) + sup = cat.get("superseded_by", []) + add(f"option_categories.{name}.superseded_by are real CLI options", + isinstance(sup, list) and all(f in flags for f in sup), + f"unknown: {[f for f in sup if f not in flags]}") + # the rho_setter concern, if present, must list all rho setters (only one may + # be active, so OOTB stacking a second would be a hard error). + if "rho_setter" in oc: + rho_setters = {"--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho"} + sup = set(oc["rho_setter"].get("superseded_by", [])) + add("option_categories.rho_setter.superseded_by lists every rho setter", + rho_setters <= sup, f"missing: {sorted(rho_setters - sup)}") + + # additional options catch-all + ao = policy.get("additional_options", {}).get("options", []) + for opt in ao: + add(f"additional_options flag {opt.get('flag')} is a real CLI option", + opt.get("flag") in flags, str(opt.get("flag"))) + sup = opt.get("superseded_by", [opt.get("flag")]) + add(f"additional_options {opt.get('flag')} superseded_by are real options", + all(f in flags for f in sup)) + + # suggestions: disabled names real generators + gen_names = {g.__name__ for g in ootb.SUGGESTION_GENERATORS} + disabled = policy.get("suggestions", {}).get("disabled", []) + add("suggestions.disabled names real generators", + isinstance(disabled, list) and all(d in gen_names for d in disabled), + f"unknown: {[d for d in disabled if d not in gen_names]}") + + # _cold_start_guess entries name real keys in their block + for block_name, block in policy.items(): + if not isinstance(block, dict) or "_cold_start_guess" not in block: + continue + guesses = block["_cold_start_guess"] + bad = [g for g in guesses if g not in block] + add(f"{block_name}._cold_start_guess entries name real keys", not bad, + f"stray: {bad}") + + # DECOMPOSITION_FLAGS line up with the real vocabulary + bad = sorted(f for f in ootb.DECOMPOSITION_FLAGS if f not in flags) + add("out_of_the_box.DECOMPOSITION_FLAGS are all real CLI options", not bad, + f"unknown: {bad}") + + return checks + + +# --------------------------------------------------------------------------- +# Layer 2: decision checks on hand-built synthetic Facts (the CI-gating subset) +# --------------------------------------------------------------------------- + + +def _facts(**kw): + """Build a Facts with sensible defaults for the synthetic decision checks.""" + base = dict(module_name="synthetic", num_ranks=3, + available_solvers={"gurobi_persistent"}, num_scens=10, + effort="base", vars_int=0, vars_cont=10, nonants_total=3, + nonants_int=0) + base.update(kw) + return ootb.Facts(**base) + + +def _bundle_arg(decision): + for a in decision.args: + if a.flag == "--scenarios-per-bundle": + return int(a.value) + return None + + +def _rho_setter_args(decision): + setters = {"--grad-rho", "--sensi-rho", "--coeff-rho", "--sep-rho"} + return [a.flag for a in decision.args if a.flag in setters] + + +def validate_decisions_synthetic(policy: dict) -> list: + checks = [] + minr = policy["ef_fallback"]["min_ranks_for_decomposition"] + + def add(name, ok, detail=""): + checks.append(Check("decision", name, ok, detail)) + + # EF when below the rank floor + d = ootb.recommend(_facts(num_ranks=minr - 1), policy) + add("below rank floor -> EF", d.run_ef and d.ef_reason == "min_ranks", + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # EF when the whole problem is tiny (base effort gate) + d = ootb.recommend(_facts(num_ranks=minr, num_scens=2, + vars_cont=1, vars_int=0), policy) + add("tiny problem above rank floor -> EF", d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # NOT EF when the whole problem is large / integer-heavy + d = ootb.recommend(_facts(num_ranks=6, num_scens=1000, + vars_cont=500, vars_int=200, nonants_int=50), policy) + add("large integer-heavy problem -> decompose (not EF)", not d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # forced decomposition wins even on a tiny problem + d = ootb.recommend(_facts(num_ranks=minr, num_scens=2, vars_cont=1, + user_flags={"--lagrangian", "--xhatshuffle"}), policy) + add("user-forced decomposition (>= rank floor) -> never EF", not d.run_ef, + f"run_ef={d.run_ef} reason={d.ef_reason}") + + # bundling validity: when OOTB bundles, spb divides num_scens & #bundles>=ranks + d = ootb.recommend(_facts(num_ranks=minr, num_scens=120, vars_cont=5, + user_flags={"--lagrangian"}), policy) + spb = _bundle_arg(d) + if spb is None: + add("bundling validity (no bundle chosen here)", True, "no --scenarios-per-bundle") + else: + ok = (120 % spb == 0) and (120 // spb >= d.intra_ranks) + add("bundling: spb divides num_scens and #bundles >= intra_ranks", ok, + f"spb={spb}, #bundles={120 // spb}, intra_ranks={d.intra_ranks}") + + # minus tier never bundles (no size profile) + d = ootb.recommend(_facts(effort="minus", vars_int=None, vars_cont=None, + nonants_total=None, nonants_int=None, + num_ranks=minr, num_scens=120, + user_flags={"--lagrangian"}), policy) + add("minus tier never bundles", _bundle_arg(d) is None, + f"spb={_bundle_arg(d)}") + + # no conflicting rho setters: the user's rho setter must suppress OOTB's + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_int=5, + vars_cont=10, nonants_int=2, + user_flags={"--lagrangian", "--sensi-rho"}), policy) + add("user rho setter suppresses OOTB's (<=0 OOTB rho setters)", + len(_rho_setter_args(d)) == 0, f"OOTB rho setters: {_rho_setter_args(d)}") + + # OOTB adds at most one rho setter on its own + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_int=5, vars_cont=10, + user_flags={"--lagrangian"}), policy) + add("OOTB adds at most one rho setter", len(_rho_setter_args(d)) <= 1, + f"OOTB rho setters: {_rho_setter_args(d)}") + + # solver routing by problem class: the chosen solver must come from the + # matching per-class preference list (base/plus, where the class is known). + by_class = policy["solver"].get("preference_order_by_class", {}) + all_solvers = set(policy["solver"]["preference_order"]) + + def _routes(name, model_degree, vars_int, available): + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_cont=10, + vars_int=vars_int, model_degree=model_degree, + available_solvers=available), policy) + want = by_class.get(d.problem_class, []) + add(f"{name} -> solver from the {d.problem_class} list", + d.chosen_solver is not None and d.chosen_solver in want, + f"class={d.problem_class} chose={d.chosen_solver} want in {want}") + + _routes("continuous linear", "linear", 0, all_solvers) + _routes("integer linear", "linear", 5, all_solvers) + _routes("continuous quadratic", "quadratic", 0, all_solvers) + _routes("integer quadratic", "quadratic", 5, all_solvers) + # more-than-quadratic continuous routes to the NLP solver, never a MIP solver + _routes("continuous nonlinear", "nonlinear", 0, {"gurobi", "ipopt"}) + + # a nonlinear model with NO NLP solver must NOT fall back to a MIP solver + d = ootb.recommend(_facts(num_ranks=6, num_scens=10, vars_cont=10, vars_int=0, + model_degree="nonlinear", + available_solvers={"gurobi", "cbc"}), policy) + add("nonlinear model without an NLP solver -> no solver (not a MIP solver)", + d.chosen_solver is None, f"chose={d.chosen_solver}") + + return checks + + +# --------------------------------------------------------------------------- +# Layer 2: decision checks on real, probe-instantiated example models (out of CI) +# --------------------------------------------------------------------------- + + +def _load_example_module(spec: dict): + if spec["dir"] not in sys.path: + sys.path.insert(0, spec["dir"]) + return importlib.import_module(spec["module"]) + + +def _example_cfg(spec: dict, module): + """Build a fully-declared cfg for an example without parsing a command line, + then set its scenario count / branching factors from spec["scen_args"].""" + cfg = config.Config() + parsing.add_driver_args(cfg, module) + s = spec["scens"] + if "num_scens" in s: + cfg.num_scens = s["num_scens"] + else: + cfg.branching_factors = list(s["branching_factors"]) + return cfg + + +def validate_decisions_examples(policy: dict) -> list: + checks = [] + for spec in example_models(): + name = spec["name"] + try: + module = _load_example_module(spec) + cfg = _example_cfg(spec, module) + except Exception as e: # noqa: BLE001 + checks.append(Check("decision", f"example {name}: set up", False, + f"{type(e).__name__}: {e}")) + continue + for nranks in (1, 3, 8): + try: + cfg.inspect_only = str(nranks) # plan as if nranks (no MPI) + facts = ootb.gather_facts(module, cfg, "base", policy) + d = ootb.recommend(facts, policy) + except Exception as e: # noqa: BLE001 + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: recommend", + False, f"{type(e).__name__}: {e}")) + continue + # the probe must have populated a size profile + prof_ok = facts.vars_cont is not None and facts.vars_int is not None + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: probe size profile", + prof_ok, + f"int={facts.vars_int} cont={facts.vars_cont} " + f"nonants={facts.nonants_total}")) + # internal consistency of the recommendation + spb = _bundle_arg(d) + bundle_ok = spb is None or (facts.num_scens % spb == 0 + and facts.num_scens // spb >= d.intra_ranks) + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: bundling valid", + bundle_ok, f"spb={spb} intra={d.intra_ranks} " + f"num_scens={facts.num_scens}")) + checks.append(Check("decision", + f"example {name} @ {nranks} ranks: <=1 rho setter", + len(_rho_setter_args(d)) <= 1, + str(_rho_setter_args(d)))) + return checks + + +# --------------------------------------------------------------------------- +# Layer 3: actually run the recommended configs (--run; slow; never a CI gate) +# --------------------------------------------------------------------------- + + +_OBJ_RE = re.compile(r"EF objective:\s*([-\d.eE+]+)") +# A termination stats row, tolerating global_toc's leading "[ 0.86] " prefix: +# [ 0.86] 99 X -124309.6133 -123177.9852 0.910% 1131.6281 +_STAT_RE = re.compile(r"^(?:\[[^\]]*\]\s*)?(\d+)\s+.*?([\d.]+)%\s+[-\d.eE+]+\s*$") + + +def _parse_decompose(out: str): + """Return (converged, iterations, rel_gap) parsed from a decomposition run's + stdout. 'Terminating based on inter-cylinder' (or 'Cylinder convergence') + marks gap convergence; otherwise the run exhausted its iterations.""" + converged = ("Terminating based on inter-cylinder" in out + or "Cylinder convergence" in out) + iters = rel_gap = None + seen_stats = False + for line in out.splitlines(): + if "Statistics at termination" in line: + seen_stats = True + continue + if seen_stats: + m = _STAT_RE.match(line) + if m: + iters = int(m.group(1)) + rel_gap = float(m.group(2)) / 100.0 + return converged, iters, rel_gap + + +def _run_one(spec, mode, nranks, extra_args, timeout): # pragma: no cover + """Run one example via generic_cylinders in a subprocess and time it.""" + if mode == "EF": + cmd = [sys.executable, "-m", "mpisppy.generic_cylinders"] + else: + cmd = ["mpiexec", "-np", str(nranks), sys.executable, "-m", "mpi4py", + "-m", "mpisppy.generic_cylinders"] + cmd += ["--module-name", spec["module"]] + _scen_cli(spec) + extra_args + rec = RunRecord(example=spec["name"], + env={"ranks": nranks, "mode": mode}, mode=mode) + start = time.time() + try: + proc = subprocess.run(cmd, cwd=spec["dir"], capture_output=True, + text=True, timeout=timeout, env=_child_env()) + rec.returncode = proc.returncode + out = proc.stdout + "\n" + proc.stderr + rec.detail = (proc.stderr.strip().splitlines() or [""])[-1] + except subprocess.TimeoutExpired: + rec.returncode = None + rec.walltime = time.time() - start + rec.flagged = True + rec.flag_reason = f"timed out after {timeout}s" + return rec + rec.walltime = time.time() - start + return rec, out + + +def validate_runs(policy_path: str, *, ef_time_limit=EF_TIME_LIMIT_SEC, + ef_gap=EF_GAP_TARGET) -> list: # pragma: no cover + """Run two configurations per example and flag the two failure modes.""" + records = [] + pol_arg = ["--out-of-the-box", policy_path] if policy_path else ["--out-of-the-box"] + for spec in example_models(): + # (a) EF: few ranks -> OOTB picks the EF. Flag if it misses the gap/time. + res = _run_one(spec, "EF", 1, ["--out-of-the-box-minus"] + ( + ["--EF-mipgap", str(ef_gap)] if spec["kind"] == "2-stage MIP" else []), + timeout=ef_time_limit) + if isinstance(res, RunRecord): # timed out + res.flag_reason = f"EF missed a {ef_gap:.0%} gap within {ef_time_limit}s" + records.append(res) + else: + rec, out = res + m = _OBJ_RE.search(out) + rec.objective = float(m.group(1)) if m else None + if rec.returncode != 0 or rec.objective is None: + rec.flagged = True + rec.flag_reason = "EF run did not complete cleanly" + records.append(rec) + + # (b) decomposition (forced via --lagrangian, to exercise that path even + # on a small problem). Flag if it maxes out iterations instead of + # converging on the inter-cylinder gap. + res = _run_one(spec, "decompose", 3, pol_arg + ["--lagrangian"], + timeout=ef_time_limit) + if isinstance(res, RunRecord): # timed out + records.append(res) + else: + rec, out = res + converged, iters, rel_gap = _parse_decompose(out) + rec.iterations, rec.rel_gap = iters, rel_gap + if rec.returncode == 0 and not converged: + rec.flagged = True + rec.flag_reason = "cylinders maxed out on iterations (no gap convergence)" + elif rec.returncode != 0: + rec.flagged = True + rec.flag_reason = "decomposition run did not complete cleanly" + records.append(rec) + return records + + +# --------------------------------------------------------------------------- +# Orchestration + report +# --------------------------------------------------------------------------- + + +def run_validation(policy_path: str, *, examples=False, run=False) -> dict: + """Run the requested layers and return a machine-readable report dict.""" + report = {"policy_file": policy_path, "policy_version": None, + "checks": [], "runs": [], "ok": True} + try: + policy = ootb.load_policy(policy_path or None) + except Exception as e: # noqa: BLE001 + report["checks"].append( + Check("static", "policy file parses", False, + f"{type(e).__name__}: {e}").as_dict()) + report["ok"] = False + return report + report["policy_version"] = policy.get("policy_version") + + checks = validate_static(policy) + validate_decisions_synthetic(policy) + if examples: + checks += validate_decisions_examples(policy) + report["checks"] = [c.as_dict() for c in checks] + report["ok"] = all(c.ok for c in checks) + + if run: + records = validate_runs(policy_path) + report["runs"] = [r.as_dict() for r in records] + + return report + + +def format_report(report: dict) -> str: + lines = [] + lines.append("=" * 72) + lines.append(f"OOTB policy validation: {report['policy_file'] or '(default)'}") + lines.append(f"policy_version: {report['policy_version']}") + lines.append("=" * 72) + + by_layer = {} + for c in report["checks"]: + by_layer.setdefault(c["layer"], []).append(c) + for layer in ("static", "decision"): + layer_checks = by_layer.get(layer, []) + if not layer_checks: + continue + npass = sum(1 for c in layer_checks if c["ok"]) + lines.append(f"\n[{layer}] {npass}/{len(layer_checks)} passed") + for c in layer_checks: + mark = "PASS" if c["ok"] else "FAIL" + suffix = f" ({c['detail']})" if c["detail"] else "" + if not c["ok"] or c["detail"]: + lines.append(f" {mark} {c['name']}{suffix}") + + failures = [c for c in report["checks"] if not c["ok"]] + if failures: + lines.append(f"\n*** {len(failures)} CHECK FAILURE(S) ***") + for c in failures: + lines.append(f" - [{c['layer']}] {c['name']}: {c['detail']}") + + if report["runs"]: + lines.append("\n" + "-" * 72) + lines.append("Layer 3 runs (recorded; not auto-judged except the flags):") + for r in report["runs"]: + wt = f"{r['walltime']:.1f}s" if r["walltime"] is not None else "?" + extra = [] + if r["objective"] is not None: + extra.append(f"obj={r['objective']:.4g}") + if r["rel_gap"] is not None: + extra.append(f"gap={r['rel_gap']:.2%}") + if r["iterations"] is not None: + extra.append(f"iters={r['iterations']}") + lines.append(f" {r['example']:8} {r['mode']:10} rc={r['returncode']} " + f"{wt:>8} {' '.join(extra)}") + flagged = [r for r in report["runs"] if r["flagged"]] + lines.append("\n*** FLAGGED RUNS (review these) ***" if flagged + else "\nNo runs flagged.") + for r in flagged: + lines.append(f" !! {r['example']} [{r['mode']}]: {r['flag_reason']}") + + lines.append("\n" + ("OVERALL: PASS" if report["ok"] else "OVERALL: FAIL")) + return "\n".join(lines) + + +def main(argv=None): + p = argparse.ArgumentParser( + prog="python -m mpisppy.generic.ootb_validate", + description="Validate an OOTB policy file (see design doc sec. 8).") + p.add_argument("policy", nargs="?", default="", + help="policy file path (default: the shipped default policy)") + p.add_argument("--examples", action="store_true", + help="also run decision checks on probe-instantiated real " + "example models (needs a solver to instantiate; not CI)") + p.add_argument("--run", action="store_true", + help="also actually RUN the recommended configs (layer 3; " + "slow, needs a solver and mpiexec; never a CI gate)") + p.add_argument("--json", metavar="PATH", default=None, + help="write the machine-readable report to PATH") + args = p.parse_args(argv) + + report = run_validation(args.policy, examples=args.examples, run=args.run) + print(format_report(report)) + if args.json: + with open(args.json, "w") as fp: + json.dump(report, fp, indent=2) + print(f"\n[wrote JSON report to {args.json}]") + return 0 if report["ok"] else 1 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(main()) diff --git a/mpisppy/generic/out_of_the_box.py b/mpisppy/generic/out_of_the_box.py new file mode 100644 index 000000000..1d7b09fc6 --- /dev/null +++ b/mpisppy/generic/out_of_the_box.py @@ -0,0 +1,845 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""Out-of-the-box (OOTB) auto-configuration. + +This module is the thin Python *interpreter* described in +``doc/designs/out_of_the_box_design.md`` (sec. 5/5.1). It turns: + + * a set of FACTS about the environment (MPI ranks, installed solvers, + optional cores/memory) and the model (scenario count, stage structure, and + -- at the base/plus tiers -- a probe size profile), and + * a dated declarative POLICY file (``ootb_policies/ootb_policy_*.json``) + +into a recommended mpi-sppy configuration, a human-readable reason for every +choice, a post-run "Suggestions" list, and the equivalent explicit command +line. The recommendation is applied back onto the ``Config`` object so the +normal ``generic_cylinders`` driver path executes it. + +Design commitments: + * USER OPTIONS ALWAYS WIN -- ``recommend()`` defers to anything in + ``facts.user_flags`` (requirement 0). + * The decision logic is plain, ordered, readable Python -- no rules engine, + no neural net. The *numbers* live in the policy file, not here. + * Every decision records WHY, so transparency (requirement 4) falls out. + +Pure vs. wired: + * Pure functions of (facts, policy): ``recommend()`` and its helpers, the + suggestion generators, ``load_policy``. Hand-build a ``Facts`` and call + ``recommend()`` to see the choices (this is what the validator's + synthetic-facts layer does). + * Wiring (touches MPI/Pyomo/Config): ``gather_facts``, ``verify_instantiation``, + ``apply_decision``, and the ``configure``/``report_suggestions`` entry + points the driver calls. +""" + +from __future__ import annotations + +import json +import math +import os +import sys +from dataclasses import dataclass, field + +# --------------------------------------------------------------------------- +# Effort tiers: the three OOTB flags map to one internal level. The flag name +# is the cfg key (dashes -> underscores); the value is an optional policy path. +# --------------------------------------------------------------------------- + +EFFORT_FLAGS = { + "out_of_the_box_minus": "minus", + "out_of_the_box": "base", + "out_of_the_box_plus": "plus", +} + + +# --------------------------------------------------------------------------- +# Data carriers +# --------------------------------------------------------------------------- + + +@dataclass +class Facts: + """Everything OOTB is allowed to look at (besides the policy).""" + + module_name: str + num_ranks: int + available_solvers: set[str] # names for which SolverFactory(.).available() + num_scens: int + effort: str = "base" # "minus" | "base" | "plus" + # probe size profile -- populated at the base/plus tiers, None at minus + vars_int: int | None = None # integer/binary vars per scenario + vars_cont: int | None = None # continuous vars per scenario + nonants_total: int | None = None # first-stage (nonanticipative) vars + nonants_int: int | None = None # integer nonants + # user-model nonlinearity from the probe: "linear" | "quadratic" | "nonlinear" + # (more nonlinear than quadratic, i.e. degree > 2 or non-polynomial); None at + # the minus tier (nothing is instantiated, so the class cannot be determined). + model_degree: str | None = None + multistage: bool = False + branching_factors: list | None = None # for the equivalent command line + user_solver_name: str | None = None # name the user gave (if any) + num_cores: int | None = None # best effort; may be None + memory_gb: float | None = None # best effort; may be None + under_slurm: bool = False + user_flags: set[str] = field(default_factory=set) # CLI flags the user set + + +@dataclass +class ChosenArg: + flag: str # e.g. "--lagrangian" or "--solver-name" + value: str | None # None => boolean flag + reason: str + + +@dataclass +class Decision: + run_ef: bool = False + ef_reason: str | None = None # "min_ranks" | "few_scens" | "user" ... + chosen_solver: str | None = None + problem_class: str | None = None # LP|MIP|QP|MIQP|NLP|MINLP (base/plus) + num_cylinders: int = 1 # hub + spokes actually configured + intra_ranks: int = 1 # widest cylinder's rank count + rank_split: dict = field(default_factory=dict) # cylinder -> ranks (flex) + args: list[ChosenArg] = field(default_factory=list) # what OOTB added + notes: list[str] = field(default_factory=list) # full reasoning trace + suggestions: list[str] = field(default_factory=list) # filled AFTER the run + + def command_line(self, facts: "Facts") -> str: + """The explicit command the OOTB choices are equivalent to (req. 4). + + Anchored with the module and scenario specification so it is runnable + without --out-of-the-box; the OOTB-added flags follow. (Options the user + set explicitly are not repeated here -- they were already on the user's + command line and OOTB left them untouched.) + """ + parts = [ + f"mpiexec -np {facts.num_ranks} python -m mpi4py -m " + f"mpisppy.generic_cylinders", + f"--module-name {facts.module_name}", + ] + if facts.multistage and facts.branching_factors: + parts.append("--branching-factors " + + " ".join(str(b) for b in facts.branching_factors)) + else: + parts.append(f"--num-scens {facts.num_scens}") + for a in self.args: + parts.append(a.flag if a.value is None else f"{a.flag} {a.value}") + return " ".join(parts) + + +# Flags that mean "the user wants a decomposition": any wired spoke or a +# non-default hub. If the user set one of these AND has >= the rank floor, OOTB +# must NOT substitute the EF, even for a small problem (requirement 0). This is +# the generic_cylinders vocabulary -- a fact, not a focus preference -- so it +# lives in code, and the validator checks it against the actual CLI flags. +DECOMPOSITION_FLAGS = frozenset({ + # wired spokes + "--lagrangian", "--fwph", "--ph-dual", "--ph-xfeas-spoke", "--relaxed-ph", + "--subgradient", "--reduced-costs", "--xhatshuffle", "--xhatxbar", + "--xhatlshaped", + # non-default hubs + "--APH", "--subgradient-hub", "--fwph-hub", "--ph-primal-hub", + "--lshaped-hub", "--cg-hub", "--dualcg-hub", +}) + + +# --------------------------------------------------------------------------- +# The interpreter: pure (facts, policy) -> Decision [COMPLETE] +# --------------------------------------------------------------------------- + + +def recommend(facts: Facts, policy: dict) -> Decision: + """Apply the policy to the facts. Ordered, each step records its reason.""" + d = Decision() + + def choose(flag: str, value: str | None, reason: str) -> bool: + # requirement 0: never override an explicit user choice. + if flag in facts.user_flags: + d.notes.append(f"{flag}: kept user's value (OOTB defers)") + return False + d.args.append(ChosenArg(flag, value, reason)) + shown = flag if value is None else f"{flag} {value}" + d.notes.append(f"{shown}: {reason}") + return True + + # --- step 1: pick the solver NAME --------------------------------------- + # The flag is emitted only once the EF gate decides: --EF-solver-name for the + # EF, --solver-name to decompose (mpi-sppy uses different keys). Here we just + # settle on the name -- routed by the model's PROBLEM CLASS (LP/MIP/QP/MIQP/ + # NLP/MINLP) so a more-than-quadratic model gets an NLP solver (ipopt) and an + # integer model never gets a continuous-only one. + sp = policy["solver"] + d.problem_class = _problem_class(facts) + if d.problem_class is not None: + d.notes.append(f"model class: {d.problem_class} ({_class_english(facts)})") + if facts.user_flags & {"--solver-name", "--EF-solver-name"} and facts.user_solver_name: + d.chosen_solver = facts.user_solver_name + d.notes.append(f"solver: kept user's value ({facts.user_solver_name}); " + f"OOTB defers") + else: + by_class = sp.get("preference_order_by_class", {}) + if d.problem_class is not None and d.problem_class in by_class: + order = by_class[d.problem_class] + why = f"first available preferred for a {d.problem_class} model" + else: + # minus tier (class unknown) or a class with no dedicated list: use + # the master preference order. + order = sp["preference_order"] + why = "first available in preference order" + d.chosen_solver = _first_available(order, facts.available_solvers) + if d.chosen_solver is None: + d.notes.append( + f"WARNING: no installed solver for this model " + f"({d.problem_class or 'class unknown'}); tried {', '.join(order)}. " + f"User must supply one (--solver-name).") + else: + d.notes.append(f"solver: {d.chosen_solver} ({why})") + + # --- step 2: EF gate ---------------------------------------------------- + # Reuses the bundle effort() model on the WHOLE problem (all scenarios as one + # model) vs an ABSOLUTE EF budget. base: effort(num_scens) <= ef_effort_budget; + # plus would use ef_target_seconds via a measured t1 (stubbed); minus (no + # size profile): the count rule. + ef = policy["ef_fallback"] + have_profile = facts.vars_cont is not None or facts.vars_int is not None + if "--EF" in facts.user_flags: + d.run_ef, d.ef_reason = True, "user" + d.notes.append("--EF: user requested") + elif facts.num_ranks < ef["min_ranks_for_decomposition"]: + d.run_ef, d.ef_reason = True, "min_ranks" + choose("--EF", None, + f"only {facts.num_ranks} ranks; decomposition needs " + f">= {ef['min_ranks_for_decomposition']}") + elif facts.user_flags & DECOMPOSITION_FLAGS: + # User explicitly asked for a decomposition and has enough ranks, so we + # never substitute the EF -- even for a small problem (requirement 0). + forced = ", ".join(sorted(facts.user_flags & DECOMPOSITION_FLAGS)) + d.notes.append(f"EF gate skipped: user requested decomposition ({forced}) " + f"with >= {ef['min_ranks_for_decomposition']} ranks") + elif have_profile: + # base/plus: EF when the whole monolith is within the absolute EF budget. + whole = _effort(facts.num_scens, facts, policy["effort_scaling"]) + if whole <= ef["ef_effort_budget"]: + d.run_ef, d.ef_reason = True, "small_effort" + choose("--EF", None, + f"whole-problem effort {whole:.0f} <= EF budget " + f"{ef['ef_effort_budget']}") + elif facts.num_scens <= ef["ef_if_num_scens_at_most"]: + # minus: no profile -> count rule + d.run_ef, d.ef_reason = True, "few_scens" + choose("--EF", None, + f"only {facts.num_scens} scenarios; too few to decompose") + + if d.run_ef: + # EF bypasses the hub/spoke system entirely; skip spokes + bundling. + # The EF path reads cfg.EF_solver_name (its own key). + if d.chosen_solver is not None: + choose("--EF-solver-name", d.chosen_solver, + f"EF solver ({d.chosen_solver})") + d.num_cylinders = 1 + return d + + # decomposition solver + (if LP/MIP-only) prox linearization. The hub/spoke + # path reads cfg.solver_name. + if d.chosen_solver is not None: + choose("--solver-name", d.chosen_solver, + f"decomposition solver ({d.chosen_solver})") + if d.chosen_solver in sp["lp_mip_only_force_linearize_prox"]: + choose("--linearize-proximal-terms", None, + f"{d.chosen_solver} is LP/MIP-only; the PH prox must be linearized") + + # --- step 3: spoke roster -- small core, widened by ranks -------------- + # Take the minimal core (>=1 outer + >=1 inner). Add further ladder rungs + # only while each cylinder keeps >= min_ranks_per_cylinder ranks (coarse, + # uniform gate) and we stay <= max_cylinders -- prefer giving cylinders + # width over piling on weaker spokes (6 ranks -> 3 cylinders, not 6). + ladder = policy["spoke_ladder"] + max_cyl = ladder["max_cylinders"] + min_rpc = policy["rank_allocation"]["min_ranks_per_cylinder"] + all_rungs = sorted(ladder["rungs"], key=lambda r: r["priority"]) + need = dict(ladder["core_roster_min"]) # e.g. {"outer": 1, "inner": 1} + chosen = [] + for r in all_rungs: # minimal core + if need.get(r["bound"], 0) > 0: + chosen.append(r) + need[r["bound"]] -= 1 + for r in all_rungs: # widen-aware additions + if r in chosen: + continue + cyl_if_added = 2 + len(chosen) # hub + chosen + this rung + if cyl_if_added > max_cyl or facts.num_ranks // cyl_if_added < min_rpc: + break + chosen.append(r) + for r in chosen: + choose(r["flag"], None, f"spoke ({r['bound']}, priority {r['priority']})") + d.num_cylinders = 1 + len(chosen) + + # --- step 4: rank allocation -- UNBALANCED by per-cylinder ratio -------- + # Split ranks across cylinders by ratio (flex-ranks), not uniformly; xhat + # cylinders are cheaper so get a smaller share. We emit the per-spoke + # ---rank-ratio flags (only when a spoke's ratio differs from the + # default) and let WheelSpinner.apportion_ranks do the real split. CRUDE + # cold-start: the real split depends on subproblem solve cost (a plus-tier + # refinement). + ra = policy["rank_allocation"] + default_ratio = ra["default_rank_ratio"] + # ratios in cylinder order: hub first (always the default), then spokes. + ratios = [default_ratio] + for r in chosen: + spoke_ratio = ra["rank_ratios"].get(r["flag"], default_ratio) + ratios.append(spoke_ratio) + if spoke_ratio != default_ratio: + choose(f"{r['flag']}-rank-ratio", _fmt_ratio(spoke_ratio), + f"flex-ranks: cheaper cylinder gets a {spoke_ratio} share " + f"(crude cold-start)") + d.intra_ranks, d.rank_split = _rank_layout(facts.num_ranks, ratios, chosen) + d.notes.append("rank split (flex-ranks, crude cold-start): " + + ", ".join(f"{k}={v}" for k, v in d.rank_split.items())) + + # --- step 5: proper bundling -- how BIG? (design "Bundle sizing") ------- + # minus CANNOT bundle: with no size profile there is no safe way to size a + # bundle, so the minus tier always runs unbundled. + bs = policy["bundle_sizing"] + if not have_profile: # have_profile computed in the EF-gate step above + d.notes.append("bundling: minus tier (no size profile); running unbundled") + elif facts.num_scens >= bs["min_scens_to_consider_bundling"]: + b_min = max(d.intra_ranks, + bs["min_bundles_per_intra_rank"] * d.intra_ranks) + spb = _pick_spb_by_effort( + facts.num_scens, b_min, facts, policy["effort_scaling"], + bs["base_max_hardness_vs_single_scenario"]) + if spb is not None: + nb = facts.num_scens // spb + choose("--scenarios-per-bundle", str(spb), + f"{facts.num_scens} scenarios -> {nb} bundles of {spb} " + f"(effort <= {bs['base_max_hardness_vs_single_scenario']}x a " + f"single scenario, >= {b_min} bundles)") + else: + d.notes.append( + f"bundling: no divisor of {facts.num_scens} qualifies " + f"(>= {b_min} bundles within budget); running unbundled" + ) + + # --- step 6: extra options by concern; any superseded_by flag defers ---- + # Per-concern override: OOTB backs off a whole concern (e.g. its rho setter) + # if the user set ANY equivalent flag, not just the identical one -- mpi-sppy + # allows only one rho setter, so stacking would be a hard error. + for name, cat in policy.get("option_categories", {}).items(): + if name.startswith("_"): + continue + if any(f in facts.user_flags for f in cat.get("superseded_by", [cat["flag"]])): + d.notes.append(f"{name}: superseded by a user option; OOTB defers") + else: + choose(cat["flag"], cat.get("value"), f"policy option ({name})") + # catch-all: per-flag override (superseded_by defaults to the flag itself) + for opt in policy.get("additional_options", {}).get("options", []): + if any(f in facts.user_flags for f in opt.get("superseded_by", [opt["flag"]])): + d.notes.append(f"{opt['flag']}: superseded by a user option; OOTB defers") + else: + choose(opt["flag"], opt.get("value"), "policy additional option") + + return d + + +def _fmt_ratio(x: float) -> str: + """Render a rank ratio without a trailing ``.0`` (so 0.2 stays 0.2).""" + return str(int(x)) if float(x).is_integer() else str(x) + + +def _first_available(order, available) -> str | None: + """First solver in `order` that is installed (present in `available`).""" + for name in order: + if name in available: + return name + return None + + +# (model_degree, has_integer_vars) -> problem class label. Used to route solver +# selection to the matching preference_order_by_class list. +_PROBLEM_CLASS = { + ("linear", False): "LP", + ("quadratic", False): "QP", + ("nonlinear", False): "NLP", + ("linear", True): "MIP", + ("quadratic", True): "MIQP", + ("nonlinear", True): "MINLP", +} + + +def _problem_class(facts: Facts) -> str | None: + """The model's problem class (LP/MIP/QP/MIQP/NLP/MINLP), or None when the + model was not instantiated (minus tier) so integrality/degree are unknown. + Integrality is `vars_int > 0`; the degree is `model_degree` -- "linear", + "quadratic", or "nonlinear" (more nonlinear than quadratic).""" + if facts.model_degree is None or facts.vars_int is None: + return None + return _PROBLEM_CLASS[(facts.model_degree, facts.vars_int > 0)] + + +def _class_english(facts: Facts) -> str: + """Plain-language basis for the problem class, e.g. "integer + quadratic".""" + integer = "integer" if (facts.vars_int or 0) > 0 else "continuous" + return f"{integer} + {facts.model_degree}" + + +def _effort(spb: int, facts: Facts, scaling: dict) -> float: + """Modeled solve effort of a bundle of `spb` scenarios, from the probe size + profile (facts.vars_cont/vars_int/nonants_int) and the policy effort_scaling + shape. Continuous ~linear, integers superlinear (int_exponent > 1), integer + nonants a fixed per-bundle coupling cost. See design "Bundle sizing".""" + cont = (facts.vars_cont or 0) * spb + nint = (facts.vars_int or 0) * spb + return (scaling["cont_coeff"] * cont + + scaling["int_weight"] * (nint ** scaling["int_exponent"]) + + scaling["int_nonant_coeff"] * (facts.nonants_int or 0)) + + +def _pick_spb_by_effort(num_scens: int, min_bundles: int, facts: Facts, + scaling: dict, max_hardness: float) -> int | None: + """base/plus sizer: the LARGEST scenarios_per_bundle that divides num_scens, + leaves >= min_bundles bundles, and keeps a bundle's modeled effort within + `max_hardness` x a single scenario (the relative, unit-free budget). The plus + tier feeds the same function a time-calibrated effective hardness. Returns + None when nothing past the degenerate spb==1 qualifies.""" + e1 = _effort(1, facts, scaling) + best = None + for spb in range(2, num_scens + 1): # spb==1 is "no bundling" + if num_scens % spb or num_scens // spb < min_bundles: + continue + if e1 > 0 and _effort(spb, facts, scaling) / e1 > max_hardness: + continue + if best is None or spb > best: # largest = max amortization + best = spb + return best + + +def _rank_layout(total: int, ratios: list, chosen: list) -> tuple: + """Return (intra_ranks, rank_split) for the chosen cylinders. + + `ratios` is hub-first, matching `[hub] + chosen`. When every ratio equals + the first (uniform), WheelSpinner uses the equal-rank split + (total // n_cyl per cylinder); otherwise it apportions by ratio + (largest-remainder, floor of one) -- we mirror that so the bundling + `#bundles >= intra_ranks` floor matches what actually runs. intra_ranks is + the widest cylinder's rank count (it governs the bundle floor).""" + names = ["(hub)"] + [r["flag"] for r in chosen] + if all(x == ratios[0] for x in ratios): + per = max(1, total // len(ratios)) + split = {nm: per for nm in names} + else: + from mpisppy.utils.rank_apportionment import apportion_ranks + counts = apportion_ranks(ratios, total) + split = dict(zip(names, counts)) + return max(split.values()), split + + +# --------------------------------------------------------------------------- +# Suggestions -- MOSTLY COMPUTED [COMPLETE] +# Each generator inspects facts/decision/run-outcome and COMPUTES a message +# (with live values) or returns None. The prose lives here in code; the policy +# only toggles (suggestions.disabled) or tunes them. This is the diagnostics +# layer -- deliberately distinct from the data-driven DECISIONS (design 5). +# Generators run in list order (priority); add outcome-based ones freely -- +# they receive the post-run `outcome`. +# --------------------------------------------------------------------------- + + +def _sg_ran_ef_few_ranks(d, facts, policy, outcome): + if d.run_ef and d.ef_reason == "min_ranks": + need = policy["ef_fallback"]["min_ranks_for_decomposition"] + return (f"Ran the monolithic EF because only {facts.num_ranks} MPI " + f"rank(s) were available; with >= {need} ranks OOTB would " + f"decompose (hub + bound spokes).") + return None + + +def _sg_no_class_solver(d, facts, policy, outcome): + # A problem class was detected but nothing installed handles it -- e.g. an + # MINLP with no baron/scip/bonmin/couenne, or an NLP with no ipopt. (Integer + # + more-than-quadratic models route to MINLP, which rarely has an installed + # solver.) recommend() already left chosen_solver None here. + if d.chosen_solver is None and d.problem_class: + listed = policy["solver"].get("preference_order_by_class", {}) \ + .get(d.problem_class, []) + names = ", ".join(listed) if listed else "none listed" + return (f"Detected a {d.problem_class} model, but none of the solvers " + f"OOTB prefers for it ({names}) are installed; install one or " + f"pass --solver-name explicitly.") + return None + + +def _sg_no_persistent_solver(d, facts, policy, outcome): + s = d.chosen_solver + if not d.run_ef and s is not None and not s.endswith("_persistent"): + return (f"Chosen solver '{s}' has no persistent interface available; " + f"'{s}_persistent' would warm-start subproblems and is usually " + f"much faster for PH.") + return None + + +def _sg_linearized_prox(d, facts, policy, outcome): + if not d.run_ef and \ + d.chosen_solver in policy["solver"]["lp_mip_only_force_linearize_prox"]: + return (f"'{d.chosen_solver}' is LP/MIP-only, so the PH prox is being " + f"linearized; a QP-capable solver (gurobi/cplex/xpress, or ipopt " + f"for continuous models) avoids the approximation.") + return None + + +def _sg_more_ranks(d, facts, policy, outcome): + cap = policy["spoke_ladder"]["max_cylinders"] + if not d.run_ef and d.num_cylinders < cap: + return (f"Only {d.num_cylinders} cylinders configured (cap {cap}); more " + f"MPI ranks would add bound-tightening spokes or intra-cylinder " + f"parallelism.") + return None + + +def _sg_minus_no_bundling(d, facts, policy, outcome): + if facts.effort == "minus" and not d.run_ef: + return ("Ran the --out-of-the-box-minus tier, which instantiates nothing " + "and therefore cannot size proper bundles; --out-of-the-box (base) " + "probes one scenario and can bundle when there are many scenarios.") + return None + + +def _sg_from_outcome(d, facts, policy, outcome): + # Outcome-based (post-run) computed suggestion; inert until the run captures + # an outcome (None in the minus/base tiers for now). + if outcome and outcome.get("converged") is False: + return (f"PH stopped at {outcome.get('iterations')} iterations with a " + f"{outcome.get('rel_gap', 0):.0%} gap; consider raising " + f"--max-iterations or adding a tighter bound spoke.") + return None + + +# generators in priority order (lower first) +SUGGESTION_GENERATORS = [ + _sg_ran_ef_few_ranks, + _sg_no_class_solver, + _sg_no_persistent_solver, + _sg_linearized_prox, + _sg_more_ranks, + _sg_minus_no_bundling, + _sg_from_outcome, +] + + +def make_suggestions(d: Decision, facts: Facts, policy: dict, + outcome: dict | None = None) -> list[str]: + """Build the post-run "Suggestions" list (req. 4) by running the computed + generators in priority order, skipping any named in suggestions.disabled. + Called AFTER the run so generators may use `outcome` (convergence, gap, + iters, time).""" + disabled = set(policy.get("suggestions", {}).get("disabled", [])) + out = [] + for gen in SUGGESTION_GENERATORS: + if gen.__name__ in disabled: + continue + msg = gen(d, facts, policy, outcome) + if msg: + out.append(msg) + return out + + +# --------------------------------------------------------------------------- +# Policy loading [COMPLETE] +# --------------------------------------------------------------------------- + + +def _policies_dir() -> str: + return os.path.join(os.path.dirname(__file__), "ootb_policies") + + +def load_policy(policy_file: str | None = None) -> dict: + """Load a policy. Default (policy_file None or empty): the newest dated file + with no focus token in ootb_policies/ (``ootb_policy_.json``). The + dated filename sorts lexically by ISO date, so the last is the newest.""" + if not policy_file: + d = _policies_dir() + # focus is conveyed by extra filename tokens (ootb_policy_quick_); + # the bare default is ootb_policy_.json -- two underscores exactly. + candidates = sorted( + f for f in os.listdir(d) + if f.startswith("ootb_policy_") and f.endswith(".json") + and f.count("_") == 2 + ) + if not candidates: + raise FileNotFoundError(f"no default OOTB policy files in {d}") + policy_file = os.path.join(d, candidates[-1]) + with open(policy_file) as fp: + return json.load(fp) + + +# --------------------------------------------------------------------------- +# Environment/model probing + apply-to-Config [the wiring] +# --------------------------------------------------------------------------- + + +def effort_and_policy(cfg) -> tuple: + """Return (effort, policy_path) from the cfg, or (None, None) if OOTB is off. + + The three tier flags are mutually exclusive; supplying more than one is an + error. ``policy_path`` is the optional value attached to the chosen flag + (empty string -> use the shipped default policy).""" + selected = [(cfg_key, tier) for cfg_key, tier in EFFORT_FLAGS.items() + if cfg.get(cfg_key) is not None] + if not selected: + return None, None + if len(selected) > 1: + names = ", ".join("--" + k.replace("_", "-") for k, _ in selected) + raise RuntimeError( + f"At most one out-of-the-box tier may be given; got {names}.") + cfg_key, tier = selected[0] + policy_path = cfg.get(cfg_key) or None # "" (bare flag) -> default + return tier, policy_path + + +def requested(cfg) -> bool: + """True if any out-of-the-box tier flag was supplied.""" + return any(cfg.get(k) is not None for k in EFFORT_FLAGS) + + +def _rank0() -> bool: + from mpisppy import MPI + return MPI.COMM_WORLD.Get_rank() == 0 + + +def _detect_num_ranks() -> int: + from mpisppy import MPI + return MPI.COMM_WORLD.Get_size() + + +def _inspect_ranks(cfg) -> int: + """Ranks to PLAN for. With --inspect-only N, use N so HPC users can get a + recommended command line for a target job size WITHOUT launching that many + ranks; otherwise the actual detected size. (N rides on --inspect-only, the + only flag that carries it; everything else -- solvers, model size -- still + comes from the real, possibly small, session.)""" + io = cfg.get("inspect_only", None) + if io not in (None, "", "detected"): + return int(io) + return _detect_num_ranks() + + +def _detect_available_solvers(candidates) -> set: + import pyomo.environ as pyo + found = set() + for name in candidates: + try: + if pyo.SolverFactory(name).available(exception_flag=False): + found.add(name) + except Exception: + pass + return found + + +def _detect_num_scens(module, cfg) -> int: + """Mirror mpisppy/generic/parsing.py::name_lists: cfg.num_scens, else the + product of the branching factors, else the module's full scenario list.""" + if cfg.get("num_scens") is not None: + return int(cfg.num_scens) + bf = cfg.get("branching_factors") + if bf is not None: + return int(math.prod(bf)) + return len(module.scenario_names_creator(None)) + + +def _user_flags() -> set: + """The set of long CLI flags the user actually typed (--flag, stripped of + any =value). This is how requirement 0 is honored: recommend() defers to any + flag in here. Parsing argv (rather than the post-parse cfg) is what lets us + tell a user-set value apart from a default -- argparse fills defaults for + everything, so the cfg alone cannot say what the user chose.""" + flags = set() + for tok in sys.argv[1:]: + if tok.startswith("--"): + flags.add(tok.split("=", 1)[0]) + return flags + + +def _model_degree(model) -> str: + """Classify the user model's nonlinearity from the probe: "linear" (every + active objective/constraint has polynomial degree <= 1), "quadratic" (max + degree == 2), or "nonlinear" (degree > 2, or non-polynomial such as + log/exp/x*y/y -- reported by Pyomo as polynomial_degree() == None). + + The probe is the RAW scenario_creator model, so the PH proximal term (a + quadratic that mpi-sppy attaches later) is absent and does not inflate the + degree -- this reflects the user's own model.""" + import pyomo.environ as pyo + max_deg = 0 + for obj in model.component_data_objects(pyo.Objective, active=True, + descend_into=True): + deg = obj.expr.polynomial_degree() if obj.expr is not None else 0 + if deg is None: + return "nonlinear" + max_deg = max(max_deg, deg) + for con in model.component_data_objects(pyo.Constraint, active=True, + descend_into=True): + body = con.body + deg = body.polynomial_degree() if body is not None else 0 + if deg is None: + return "nonlinear" + max_deg = max(max_deg, deg) + if max_deg > 2: + return "nonlinear" + return "quadratic" if max_deg == 2 else "linear" + + +def _size_profile(model) -> dict: + """Read a built scenario's size profile: integer vs continuous variable + counts, nonant (first-stage) counts, and the model degree (linear/quadratic/ + nonlinear). Used by the base/plus probe and by --inspect-only's + instantiation check.""" + import pyomo.environ as pyo + vars_int = vars_cont = 0 + for v in model.component_data_objects(pyo.Var, active=True, descend_into=True): + if v.is_continuous(): + vars_cont += 1 + else: + vars_int += 1 + nonants_total = nonants_int = 0 + for node in getattr(model, "_mpisppy_node_list", []): + for v in node.nonant_vardata_list: + nonants_total += 1 + if not v.is_continuous(): + nonants_int += 1 + return {"vars_int": vars_int, "vars_cont": vars_cont, + "nonants_total": nonants_total, "nonants_int": nonants_int, + "model_degree": _model_degree(model)} + + +def _build_probe_scenario(module, cfg): + """Instantiate a single RAW scenario (no bundle/ADMM/cvar wrapper) from the + model module so OOTB can measure per-scenario size. Raises if the model + cannot build.""" + names = module.scenario_names_creator(1) + kwargs = module.kw_creator(cfg) + return module.scenario_creator(names[0], **kwargs) + + +def verify_instantiation(module, cfg) -> dict: + """Build a single scenario to confirm the model instantiates -- a cheap + model smoke-test -- and return its size profile. SHARED CODE: used by the + base/plus probe in gather_facts and by --inspect-only when given WITHOUT + --out-of-the-box (the standalone model smoke-test).""" + model = _build_probe_scenario(module, cfg) + return _size_profile(model) + + +def gather_facts(module, cfg, effort: str, policy: dict) -> Facts: + """Assemble Facts from the environment, the model module, and cfg. The + `effort` tier sets how deep we look: minus reads structure only; base/plus + also instantiate `probe_scenarios` scenario(s) for the size profile.""" + solvers = _detect_available_solvers(policy["solver"]["preference_order"]) + bf = cfg.get("branching_factors") + facts = Facts( + module_name=cfg.get("module_name", "") or "", + num_ranks=_inspect_ranks(cfg), + available_solvers=solvers, + num_scens=_detect_num_scens(module, cfg), + effort=effort, + multistage=bf is not None, + branching_factors=list(bf) if bf is not None else None, + user_solver_name=cfg.get("solver_name") or cfg.get("EF_solver_name"), + num_cores=os.cpu_count(), + under_slurm=("SLURM_JOB_ID" in os.environ), + user_flags=_user_flags(), + ) + if effort in ("base", "plus"): + # one probe scenario (discarded) feeds the size-aware decisions; the + # real, layout-correct instantiation happens later in the driver. + profile = verify_instantiation(module, cfg) + facts.vars_int = profile["vars_int"] + facts.vars_cont = profile["vars_cont"] + facts.nonants_total = profile["nonants_total"] + facts.nonants_int = profile["nonants_int"] + facts.model_degree = profile["model_degree"] + # plus tier (PR2): instantiate all + brief timed solve for solve-time facts. + return facts + + +def apply_decision(decision: Decision, cfg) -> None: + """Apply the recommended args onto the Config so the normal driver path runs + the chosen configuration. User-set flags are already absent from + decision.args (recommend() never adds them), so this never overrides the + user (requirement 0). Pyomo coerces string values through each option's + domain.""" + if decision.run_ef: + cfg["EF"] = True + for a in decision.args: + key = a.flag[2:].replace("-", "_") # "--solver-name" -> "solver_name" + cfg[key] = True if a.value is None else a.value + + # The EF path reads cfg.EF_solver_name, the decomposition path cfg.solver_name + # -- two different keys. A naive user (and OOTB itself) naturally sets + # --solver-name; if OOTB then runs the EF, carry that name over so the EF + # path is never left without a solver. (recommend() already emits + # --EF-solver-name; this guarantees it independent of how cfg was populated.) + if decision.run_ef and cfg.get("EF_solver_name") is None \ + and cfg.get("solver_name") is not None: + cfg["EF_solver_name"] = cfg["solver_name"] + + +@dataclass +class OOTBState: + decision: Decision + facts: Facts + policy: dict + effort: str + + +def configure(module, cfg) -> OOTBState: + """Top-level OOTB entry called by the driver right after arg parsing. Probe + the environment/model, recommend a configuration, print it and the + equivalent command line (rank 0), and APPLY it onto cfg so the normal driver + path runs it. Returns the state so the driver can print Suggestions after + the run (see report_suggestions).""" + effort, policy_path = effort_and_policy(cfg) + policy = load_policy(policy_path) + facts = gather_facts(module, cfg, effort, policy) + decision = recommend(facts, policy) + + if _rank0(): + print(f"[out-of-the-box] tier '{effort}', policy " + f"{policy.get('policy_version', '?')}") + for note in decision.notes: + print(f" - {note}") + print("[out-of-the-box] equivalent command line:\n " + + decision.command_line(facts)) + + apply_decision(decision, cfg) + return OOTBState(decision=decision, facts=facts, policy=policy, effort=effort) + + +def report_suggestions(state: OOTBState, outcome: dict | None = None) -> None: + """Print the prioritized "Suggestions" list (req. 4), AFTER the run so it can + reflect how the run went (via `outcome`). Rank 0 only.""" + if not _rank0(): + return + state.decision.suggestions = make_suggestions( + state.decision, state.facts, state.policy, outcome) + if state.decision.suggestions: + print("[out-of-the-box] Suggestions:") + for s in state.decision.suggestions: + print(f" * {s}") + + +def inspect_only_standalone(module, cfg) -> None: + """--inspect-only WITHOUT any --out-of-the-box tier: verify one scenario can + be instantiated (a cheap model smoke-test), report its size, and stop. Rank + 0 prints; all ranks build (cheap, deterministic).""" + profile = verify_instantiation(module, cfg) + if _rank0(): + print("[inspect-only] model instantiates; one-scenario size profile:") + for k, v in profile.items(): + print(f" {k}: {v}") + print("[inspect-only] no out-of-the-box tier given; " + "skipping the production run.") diff --git a/mpisppy/generic/parsing.py b/mpisppy/generic/parsing.py index 85e85cf67..1f3a7213d 100644 --- a/mpisppy/generic/parsing.py +++ b/mpisppy/generic/parsing.py @@ -95,6 +95,21 @@ def load_module(model_fname): def parse_args(m): """Parse CLI args given the model module m. Returns a Config object.""" cfg = config.Config() + add_driver_args(cfg, m) + cfg.parse_command_line(f"mpi-sppy for {cfg.module_name}") + + cfg.checker() # looks for inconsistencies + return cfg + + +def add_driver_args(cfg, m=None): + """Declare every generic_cylinders option on cfg, WITHOUT parsing. + + Split out of parse_args so the same authoritative set of options can be + obtained without a command line -- the OOTB validator uses it (with m=None) + to learn which CLI flags are real. When m is given, its inparser_adder runs + too (the model-specific options). + """ cfg.proper_bundle_config() cfg.pickle_scenarios_config() cfg.pre_pickle_args() @@ -104,7 +119,8 @@ def parse_args(m): domain=str, default=None, argparse=True) - assert hasattr(m, "inparser_adder"), "The model file must have an inparser_adder function" + assert m is None or hasattr(m, "inparser_adder"), \ + "The model file must have an inparser_adder function" cfg.add_to_config(name="solution_base_name", description="The string used for a directory of ouput along with a csv and an npv file (default None, which means no soltion output)", domain=str, @@ -114,7 +130,8 @@ def parse_args(m): domain=str, default=None) - m.inparser_adder(cfg) + if m is not None: + m.inparser_adder(cfg) # many models, e.g., farmer, need num_scens_required # in which case, it should go in the inparser_adder function # cfg.num_scens_required() @@ -172,15 +189,11 @@ def parse_args(m): # TBD - think about adding directory for json options files cfg.mmw_args() + cfg.ootb_args() from mpisppy.generic.admm import admm_args admm_args(cfg) - cfg.parse_command_line(f"mpi-sppy for {cfg.module_name}") - - cfg.checker() # looks for inconsistencies - return cfg - def name_lists(module, cfg, bundle_wrapper=None): """Build all_scenario_names and all_nodenames from module and cfg. diff --git a/mpisppy/generic_cylinders.py b/mpisppy/generic_cylinders.py index d8d8ab832..3beacd48f 100644 --- a/mpisppy/generic_cylinders.py +++ b/mpisppy/generic_cylinders.py @@ -45,6 +45,21 @@ if hasattr(module, "get_mpisppy_helper_object"): module = module.get_mpisppy_helper_object(cfg) + # Out-of-the-box auto-configuration (and the OOTB-independent --inspect-only + # dry run). configure() probes the environment + model, prints the chosen + # configuration and equivalent command line, and mutates cfg so the normal + # driver path below runs it. --inspect-only stops before the production run. + from mpisppy.generic import out_of_the_box as ootb + ootb_state = None + if ootb.requested(cfg): # pragma: no cover (CLI entrypoint; configure() is unit-tested) + ootb_state = ootb.configure(module, cfg) + if cfg.get("inspect_only") is not None: + ootb.report_suggestions(ootb_state) # config-time suggestions only + sys.exit(0) + elif cfg.get("inspect_only") is not None: # pragma: no cover (CLI entrypoint) + ootb.inspect_only_standalone(module, cfg) + sys.exit(0) + bundle_wrapper = None # the default if proper_bundles(cfg): # Nonant name validation will fail with proper bundles because @@ -145,3 +160,8 @@ def scenario_denouement(rank, sname, s): scenario_denouement, bundle_wrapper=bundle_wrapper) if mmw_requested(cfg): do_mmw(fname, cfg, wheel=wheel) + + # Out-of-the-box: the prioritized "Suggestions" list is printed AFTER the + # run so it can also reflect how the run went (req. 4). + if ootb_state is not None: # pragma: no cover (CLI entrypoint) + ootb.report_suggestions(ootb_state) diff --git a/mpisppy/tests/test_ootb_calibrate.py b/mpisppy/tests/test_ootb_calibrate.py new file mode 100644 index 000000000..e9bd94cea --- /dev/null +++ b/mpisppy/tests/test_ootb_calibrate.py @@ -0,0 +1,107 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI test for the OOTB effort-calibration tool's PURE parts. + +Only the fitting / policy-assembly logic is exercised here -- it is solver-free. +The measurement (timed example solves) needs a solver and is run on demand / +locally, not in CI. See doc/designs/out_of_the_box_design.md sec. 9. +""" + +import unittest + +from mpisppy.generic import ootb_calibrate as cal +from mpisppy.generic import ootb_validate as val +from mpisppy.generic import out_of_the_box as ootb + + +def _synthetic_points(cont_coeff, int_weight, exponent, int_nonant_coeff): + """Points generated from a known effort model (zero measurement noise).""" + pts = [] + for (vc, vi, ni) in [(10, 0, 0), (5, 3, 2), (0, 8, 5), (20, 1, 1)]: + for spb in (1, 2, 4, 8): + cont = vc * spb + nint = vi * spb + seconds = (cont_coeff * cont + int_weight * (nint ** exponent) + + int_nonant_coeff * ni) + pts.append({"vars_cont": vc, "vars_int": vi, "nonants_int": ni, + "spb": spb, "seconds": seconds}) + return pts + + +class TestFit(unittest.TestCase): + def test_recovers_known_coefficients(self): + truth = dict(cont_coeff=0.01, int_weight=0.5, exponent=2.0, + int_nonant_coeff=0.2) + fit = cal.fit_effort_model(_synthetic_points(**truth)) + self.assertAlmostEqual(fit["cont_coeff"], 0.01, places=5) + self.assertAlmostEqual(fit["int_weight"], 0.5, places=5) + self.assertEqual(fit["int_exponent"], 2.0) + self.assertAlmostEqual(fit["int_nonant_coeff"], 0.2, places=5) + self.assertGreater(fit["r2"], 0.999) + self.assertEqual(fit["seconds_per_effort_unit"], 1.0) + + def test_requires_minimum_points(self): + with self.assertRaises(ValueError): + cal.fit_effort_model([{"vars_cont": 1, "vars_int": 0, + "nonants_int": 0, "spb": 1, "seconds": 1.0}]) + + def test_round_sig_preserves_tiny_values(self): + # decimal rounding would zero this; significant-figure rounding keeps it. + self.assertAlmostEqual(cal._round_sig(1.5618e-09), 1.5618e-09) + self.assertEqual(cal._round_sig(0.0), 0.0) + self.assertAlmostEqual(cal._round_sig(123456.789, 3), 123000.0) + + +class TestCalibratedPolicy(unittest.TestCase): + def setUp(self): + self.base = ootb.load_policy() + self.fit = cal.fit_effort_model(_synthetic_points( + cont_coeff=0.01, int_weight=0.5, exponent=2.0, int_nonant_coeff=0.2)) + + def test_effort_scaling_replaced(self): + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + es = pol["effort_scaling"] + self.assertAlmostEqual(es["cont_coeff"], 0.01, places=5) + self.assertEqual(es["int_exponent"], 2.0) + self.assertIn("seconds_per_effort_unit", es) + self.assertNotIn("_cold_start_guess", es) # numbers are now data-tuned + self.assertEqual(pol["policy_version"], "2026-07-01") + + def test_ef_budget_is_seconds_like(self): + # budget = target seconds / seconds_per_effort_unit (~1), so it reads as + # seconds rather than an opaque huge number. + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + self.assertEqual(pol["ef_fallback"]["ef_effort_budget"], + pol["ef_fallback"]["ef_target_seconds"]) + + def test_calibrated_policy_passes_static_validation(self): + # end-to-end: a fitted policy must still be well-formed. + pol = cal.calibrated_policy(self.base, self.fit, [], "gurobi", "2026-07-01") + fails = [c for c in val.validate_static(pol) if not c.ok] + self.assertEqual([], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + fails = [c for c in val.validate_decisions_synthetic(pol) if not c.ok] + self.assertEqual([], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + +class TestCalibrationSpecs(unittest.TestCase): + def test_specs_use_larger_counts(self): + specs = cal._calibration_specs() + self.assertEqual({s["name"] for s in specs}, + {"farmer", "sizes", "aircond"}) + farmer = next(s for s in specs if s["name"] == "farmer") + self.assertEqual(farmer["scens"], {"num_scens": 16}) + self.assertEqual(cal._design_columns( + {"vars_cont": 2, "vars_int": 3, "nonants_int": 4, "spb": 5}, 2.0), + [10, (15) ** 2.0, 4]) + + +if __name__ == "__main__": + unittest.main() diff --git a/mpisppy/tests/test_ootb_validate.py b/mpisppy/tests/test_ootb_validate.py new file mode 100644 index 000000000..7b4572c17 --- /dev/null +++ b/mpisppy/tests/test_ootb_validate.py @@ -0,0 +1,205 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI gate for the out-of-the-box (OOTB) policy-file validator. + +Runs only the solver-free, fast layers -- layer 1 (static schema) and the +synthetic-facts subset of layer 2 (pure recommend() decisions on hand-built +Facts) -- on the shipped policy file(s). Example instantiation, anything needing +a solver, and all of layer 3 run nightly / on demand / locally, NOT here. See +doc/designs/out_of_the_box_design.md sec. 8. +""" + +import copy +import os +import unittest + +from mpisppy.generic import ootb_validate as val +from mpisppy.generic import out_of_the_box as ootb + + +def _shipped_policy_files(): + d = ootb._policies_dir() + return [os.path.join(d, f) for f in sorted(os.listdir(d)) + if f.startswith("ootb_policy_") and f.endswith(".json")] + + +class TestStaticAndSyntheticGate(unittest.TestCase): + """Every shipped policy must pass layers 1 + 2-synthetic cleanly.""" + + def test_every_shipped_policy_passes(self): + files = _shipped_policy_files() + self.assertTrue(files, "no shipped OOTB policy files found") + for path in files: + with self.subTest(policy=os.path.basename(path)): + report = val.run_validation(path) # layers 1 + 2-synthetic only + fails = [c for c in report["checks"] if not c["ok"]] + self.assertTrue( + report["ok"], + msg="validation failures:\n" + "\n".join( + f" [{c['layer']}] {c['name']}: {c['detail']}" for c in fails)) + + def test_default_policy_static_layer(self): + policy = ootb.load_policy() + fails = [c for c in val.validate_static(policy) if not c.ok] + self.assertEqual( + [], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + def test_default_policy_decision_layer(self): + policy = ootb.load_policy() + fails = [c for c in val.validate_decisions_synthetic(policy) if not c.ok] + self.assertEqual( + [], fails, + msg="\n".join(f"{c.name}: {c.detail}" for c in fails)) + + +class TestValidFlags(unittest.TestCase): + def test_known_flags_are_valid(self): + flags = val.valid_flags() + for f in ("--lagrangian", "--xhatshuffle", "--fwph", "--subgradient", + "--reduced-costs", "--xhatxbar", "--solver-name", + "--EF-solver-name", "--scenarios-per-bundle", "--grad-rho", + "--default-rho", "--rel-gap", "--max-iterations", + "--dynamic-rho-primal-crit", "--linearize-proximal-terms"): + self.assertIn(f, flags) + + def test_decomposition_flags_are_real(self): + flags = val.valid_flags() + bad = sorted(f for f in ootb.DECOMPOSITION_FLAGS if f not in flags) + self.assertEqual([], bad, msg=f"unknown DECOMPOSITION_FLAGS: {bad}") + + +class TestValidatorCatchesBadPolicies(unittest.TestCase): + """The validator must actually FAIL on malformed policies (not just pass).""" + + def setUp(self): + self.policy = copy.deepcopy(ootb.load_policy()) + + def _fails(self, checks): + return [c for c in checks if not c.ok] + + def test_missing_top_level_key(self): + del self.policy["solver"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_bogus_option_flag(self): + self.policy["option_categories"]["termination"]["flag"] = "--not-a-real-flag" + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_spoke_rung_not_a_real_flag(self): + self.policy["spoke_ladder"]["rungs"][0]["flag"] = "--bogus-spoke" + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_cold_start_guess_names_unknown_key(self): + # regression: a _cold_start_guess entry that is prose, not a real key. + # (bundle_sizing stays hand-authored even after effort calibration.) + self.policy["bundle_sizing"]["_cold_start_guess"].append("not_a_key") + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_rho_setter_must_list_all_setters(self): + self.policy["option_categories"]["rho_setter"]["superseded_by"] = ["--grad-rho"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_class_list_with_stray_solver(self): + # a per-class list must be a subset of preference_order + self.policy["solver"]["preference_order_by_class"]["NLP"] = ["not_a_solver"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_missing_a_problem_class(self): + del self.policy["solver"]["preference_order_by_class"]["MINLP"] + self.assertTrue(self._fails(val.validate_static(self.policy))) + + def test_decision_layer_detects_broken_ef_gate(self): + # If the rank floor is absurdly high, OOTB would never decompose; the + # "large problem -> decompose" decision check must then fail. + self.policy["ef_fallback"]["min_ranks_for_decomposition"] = 10**9 + self.assertTrue(self._fails(val.validate_decisions_synthetic(self.policy))) + + +class TestExamplesLayer(unittest.TestCase): + """The decision-on-real-models layer builds scenarios (no solve), so it runs + here; the farmer checks must pass.""" + + def test_validate_decisions_examples_farmer_passes(self): + checks = val.validate_decisions_examples(ootb.load_policy()) + self.assertTrue(checks) + farmer = [c for c in checks if "farmer" in c.name] + self.assertTrue(farmer) + self.assertTrue(all(c.ok for c in farmer), + msg="\n".join(f"{c.name}: {c.detail}" + for c in farmer if not c.ok)) + + +class TestHelpers(unittest.TestCase): + def test_scen_cli(self): + self.assertEqual(val._scen_cli({"scens": {"num_scens": 6}}), + ["--num-scens", "6"]) + self.assertEqual( + val._scen_cli({"scens": {"branching_factors": [3, 2]}}), + ["--branching-factors", "3 2"]) # one space-joined token + + def test_child_env_scrubs_mpi_vars(self): + os.environ["OMPI_TESTVAR"] = "1" + try: + self.assertNotIn("OMPI_TESTVAR", val._child_env()) + self.assertIn("PATH", val._child_env()) + finally: + del os.environ["OMPI_TESTVAR"] + + def test_example_models_paths_exist(self): + for spec in val.example_models(): + self.assertTrue(os.path.isdir(spec["dir"]), spec["dir"]) + + def test_parse_decompose_converged(self): + out = ("[ 0.8] Terminating based on inter-cylinder relative gap 0.9%\n" + "[ 0.8] Statistics at termination\n" + "[ 0.8] Iter. Best Bound Best Incumbent Rel. Gap Abs. Gap\n" + "[ 0.8] 99 -124.0 -123.0 0.910% 1.1\n") + converged, iters, gap = val._parse_decompose(out) + self.assertTrue(converged) + self.assertEqual(iters, 99) + self.assertAlmostEqual(gap, 0.0091, places=4) + + def test_parse_decompose_maxed_out(self): + out = ("[ 9.0] Statistics at termination\n" + "[ 9.0] 100 -10.0 -5.0 50.000% 5.0\n") + converged, iters, gap = val._parse_decompose(out) + self.assertFalse(converged) + self.assertEqual(iters, 100) + + +class TestReportAndMain(unittest.TestCase): + def test_format_report_covers_branches(self): + report = { + "policy_file": "p.json", "policy_version": "2026-06-28", "ok": False, + "checks": [{"layer": "static", "name": "a", "ok": True, "detail": ""}, + {"layer": "decision", "name": "b", "ok": False, + "detail": "boom"}], + "runs": [{"example": "farmer", "mode": "EF", "returncode": 0, + "walltime": 0.5, "objective": -1.0, "rel_gap": None, + "iterations": None, "flagged": False, "flag_reason": ""}, + {"example": "sizes", "mode": "decompose", "returncode": 0, + "walltime": 1.0, "objective": None, "rel_gap": 0.5, + "iterations": 100, "flagged": True, + "flag_reason": "maxed out"}], + } + text = val.format_report(report) + self.assertIn("CHECK FAILURE", text) + self.assertIn("FLAGGED RUNS", text) + self.assertIn("OVERALL: FAIL", text) + + def test_main_passes_on_default_policy(self): + self.assertEqual(val.main([]), 0) # layers 1+2 on default + + def test_main_fails_on_missing_policy(self): + self.assertEqual(val.main(["/no/such/policy.json"]), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/mpisppy/tests/test_out_of_the_box.py b/mpisppy/tests/test_out_of_the_box.py new file mode 100644 index 000000000..bf2269ee0 --- /dev/null +++ b/mpisppy/tests/test_out_of_the_box.py @@ -0,0 +1,322 @@ +############################################################################### +# mpi-sppy: MPI-based Stochastic Programming in PYthon +# +# Copyright (c) 2024, Lawrence Livermore National Security, LLC, Alliance for +# Sustainable Energy, LLC, The Regents of the University of California, et al. +# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md for +# full copyright and license information. +############################################################################### +"""CI tests for the out-of-the-box (OOTB) interpreter wiring. + +Solver-free: the OOTB decision/apply path and the base-tier probe only need to +*build* a scenario (no solve), so the whole gather_facts -> recommend -> +apply_decision -> configure flow runs in CI on the farmer example. The +environment rank count is supplied via --inspect-only N so the EF and +decomposition branches are both exercised deterministically without launching +mpiexec. (The solver-dependent run/measurement tiers are exercised on demand / +locally; see test_ootb_validate / test_ootb_calibrate.) +""" + +import os +import sys +import unittest + +import pyomo.environ as pyo + +import mpisppy.utils.config as config +from mpisppy.generic import parsing +from mpisppy.generic import out_of_the_box as ootb + + +def _repo_root(): + here = os.path.dirname(os.path.abspath(__file__)) + return os.path.dirname(os.path.dirname(here)) + + +_FARMER_DIR = os.path.join(_repo_root(), "examples", "farmer") + + +def _farmer_module(): + if _FARMER_DIR not in sys.path: + sys.path.insert(0, _FARMER_DIR) + import farmer + return farmer + + +def _farmer_cfg(num_scens=6, **overrides): + module = _farmer_module() + cfg = config.Config() + parsing.add_driver_args(cfg, module) + cfg.num_scens = num_scens + cfg.module_name = "farmer" + for k, v in overrides.items(): + cfg[k] = v + return cfg, module + + +class TestConfigureNoSolver(unittest.TestCase): + """Drive configure() (probe + recommend + apply) without solving.""" + + def setUp(self): + self._argv = sys.argv + sys.argv = ["prog", "--module-name", "farmer", "--num-scens", "6"] + + def tearDown(self): + sys.argv = self._argv + + def test_base_few_ranks_picks_ef(self): + cfg, module = _farmer_cfg(out_of_the_box="", inspect_only="1") + state = ootb.configure(module, cfg) + self.assertTrue(state.decision.run_ef) + self.assertEqual(state.decision.ef_reason, "min_ranks") + self.assertTrue(cfg.EF) # apply_decision set it + # the probe populated a size profile (farmer is continuous) + self.assertIsNotNone(state.facts.vars_cont) + self.assertEqual(state.facts.vars_int, 0) + + def test_minus_tier_decomposes_by_count(self): + cfg, module = _farmer_cfg(out_of_the_box_minus="", inspect_only="6") + state = ootb.configure(module, cfg) + self.assertFalse(state.decision.run_ef) + self.assertTrue(cfg.lagrangian and cfg.xhatshuffle) + self.assertIsNone(state.facts.vars_cont) # minus: no probe + # minus cannot bundle + self.assertIsNone(cfg.get("scenarios_per_bundle")) + + def test_base_forced_decomposition_bundles(self): + # user --lagrangian forces decomposition even though the problem is small + sys.argv = sys.argv + ["--lagrangian"] + cfg, module = _farmer_cfg(num_scens=60, out_of_the_box="", + inspect_only="3", lagrangian=True) + state = ootb.configure(module, cfg) + self.assertFalse(state.decision.run_ef) + spb = cfg.get("scenarios_per_bundle") + self.assertIsNotNone(spb) # 60 scens -> bundles + self.assertEqual(60 % int(spb), 0) + + def test_report_suggestions_runs(self): + cfg, module = _farmer_cfg(out_of_the_box="", inspect_only="1") + state = ootb.configure(module, cfg) + ootb.report_suggestions(state) # config-time suggestions + self.assertIsInstance(state.decision.suggestions, list) + + +class TestInspectStandalone(unittest.TestCase): + def test_verify_instantiation_and_standalone(self): + cfg, module = _farmer_cfg() + profile = ootb.verify_instantiation(module, cfg) + self.assertEqual(set(profile), + {"vars_int", "vars_cont", "nonants_total", "nonants_int", + "model_degree"}) + self.assertGreater(profile["vars_cont"], 0) + self.assertEqual(profile["model_degree"], "linear") # farmer is an LP + ootb.inspect_only_standalone(module, cfg) # prints, returns None + + +class TestApplyDecision(unittest.TestCase): + def test_apply_sets_cfg_values(self): + cfg, _ = _farmer_cfg() + d = ootb.Decision() + d.args = [ootb.ChosenArg("--lagrangian", None, "x"), + ootb.ChosenArg("--solver-name", "gurobi", "x"), + ootb.ChosenArg("--scenarios-per-bundle", "10", "x"), + ootb.ChosenArg("--rel-gap", "0.01", "x")] + ootb.apply_decision(d, cfg) + self.assertTrue(cfg.lagrangian) + self.assertEqual(cfg.solver_name, "gurobi") + self.assertEqual(cfg.scenarios_per_bundle, 10) + self.assertAlmostEqual(cfg.rel_gap, 0.01) + + def test_ef_solver_name_carried_over(self): + cfg, _ = _farmer_cfg() + d = ootb.Decision(run_ef=True) + d.args = [ootb.ChosenArg("--solver-name", "cplex", "x")] + ootb.apply_decision(d, cfg) + self.assertTrue(cfg.EF) + self.assertEqual(cfg.EF_solver_name, "cplex") # belt-and-suspenders + + +class TestCommandLineAndFlags(unittest.TestCase): + def test_command_line_two_stage(self): + facts = ootb.Facts("farmer", 3, set(), 6) + d = ootb.Decision() + d.args = [ootb.ChosenArg("--lagrangian", None, "x"), + ootb.ChosenArg("--solver-name", "gurobi", "x")] + cl = d.command_line(facts) + self.assertIn("--module-name farmer", cl) + self.assertIn("--num-scens 6", cl) + self.assertIn("--lagrangian", cl) + self.assertIn("-np 3", cl) + + def test_command_line_multistage(self): + facts = ootb.Facts("aircond", 3, set(), 6, multistage=True, + branching_factors=[3, 2]) + cl = ootb.Decision().command_line(facts) + self.assertIn("--branching-factors 3 2", cl) + self.assertNotIn("--num-scens", cl) + + def test_requested_and_effort_and_policy(self): + cfg, _ = _farmer_cfg(out_of_the_box="") + self.assertTrue(ootb.requested(cfg)) + effort, path = ootb.effort_and_policy(cfg) + self.assertEqual(effort, "base") + self.assertIsNone(path) + cfg2, _ = _farmer_cfg() + self.assertFalse(ootb.requested(cfg2)) + self.assertEqual(ootb.effort_and_policy(cfg2), (None, None)) + + def test_two_tiers_is_an_error(self): + cfg, _ = _farmer_cfg(out_of_the_box="", out_of_the_box_minus="") + with self.assertRaises(RuntimeError): + ootb.effort_and_policy(cfg) + + def test_user_flags_from_argv(self): + saved = sys.argv + try: + sys.argv = ["p", "--lagrangian", "--max-iterations=50", "-q"] + flags = ootb._user_flags() + self.assertIn("--lagrangian", flags) + self.assertIn("--max-iterations", flags) # =value stripped + self.assertNotIn("-q", flags) # single dash ignored + finally: + sys.argv = saved + + +class TestSuggestionGenerators(unittest.TestCase): + """Exercise each computed suggestion generator.""" + + def setUp(self): + self.policy = ootb.load_policy() + + def _msgs(self, d, facts, outcome=None): + return ootb.make_suggestions(d, facts, self.policy, outcome) + + def test_few_ranks_and_minus(self): + facts = ootb.Facts("m", 1, set(), 3, effort="minus") + d = ootb.Decision(run_ef=True, ef_reason="min_ranks") + self.assertTrue(any("only 1 MPI" in m for m in self._msgs(d, facts))) + + def test_no_persistent_and_more_ranks(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + msgs = self._msgs(d, facts) + self.assertTrue(any("persistent" in m for m in msgs)) + self.assertTrue(any("cylinders configured" in m for m in msgs)) + + def test_linearized_prox(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="cbc", num_cylinders=3) + self.assertTrue(any("linearized" in m for m in self._msgs(d, facts))) + + def test_minus_no_bundling(self): + facts = ootb.Facts("m", 6, set(), 100, effort="minus") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + self.assertTrue(any("minus" in m for m in self._msgs(d, facts))) + + def test_outcome_based(self): + facts = ootb.Facts("m", 6, set(), 10, effort="base") + d = ootb.Decision(run_ef=False, chosen_solver="gurobi", num_cylinders=3) + outcome = {"converged": False, "iterations": 100, "rel_gap": 0.2} + self.assertTrue(any("iterations" in m for m in self._msgs(d, facts, outcome))) + + def test_disabled_generator_skipped(self): + facts = ootb.Facts("m", 1, set(), 3, effort="base") + d = ootb.Decision(run_ef=True, ef_reason="min_ranks") + pol = ootb.load_policy() + pol["suggestions"]["disabled"] = ["_sg_ran_ef_few_ranks"] + msgs = ootb.make_suggestions(d, facts, pol) + self.assertFalse(any("only 1 MPI" in m for m in msgs)) + + def test_no_class_solver_suggestion(self): + # MINLP with nothing installed -> "install one or pass --solver-name" + facts = ootb.Facts("m", 6, {"gurobi"}, 10, effort="base", + vars_int=3, vars_cont=5, model_degree="nonlinear") + d = ootb.Decision(run_ef=False, chosen_solver=None, problem_class="MINLP") + self.assertTrue(any("MINLP" in m for m in self._msgs(d, facts))) + + +def _tiny_model(kind): + """A one-scenario-shaped model whose objective/constraint degree we control.""" + m = pyo.ConcreteModel() + m.x = pyo.Var(bounds=(0, 10)) + m.y = pyo.Var(bounds=(0, 10)) + m.c = pyo.Constraint(expr=m.x + m.y <= 5) # linear unless overridden + if kind == "linear": + m.o = pyo.Objective(expr=m.x + 2 * m.y) + elif kind == "quadratic": + m.o = pyo.Objective(expr=m.x ** 2 + m.y) + elif kind == "cubic": + m.o = pyo.Objective(expr=m.x ** 3 + m.y) + elif kind == "nonpoly": + m.o = pyo.Objective(expr=pyo.log(m.x + 1) + m.y) + elif kind == "quad_constraint": + m.o = pyo.Objective(expr=m.x + m.y) + m.c2 = pyo.Constraint(expr=m.x * m.y <= 4) # degree comes from a con + return m + + +class TestModelDegreeAndClass(unittest.TestCase): + def test_model_degree(self): + self.assertEqual(ootb._model_degree(_tiny_model("linear")), "linear") + self.assertEqual(ootb._model_degree(_tiny_model("quadratic")), "quadratic") + self.assertEqual(ootb._model_degree(_tiny_model("cubic")), "nonlinear") + self.assertEqual(ootb._model_degree(_tiny_model("nonpoly")), "nonlinear") + # a quadratic CONSTRAINT (linear objective) still makes the model QP + self.assertEqual(ootb._model_degree(_tiny_model("quad_constraint")), + "quadratic") + + def test_problem_class_mapping(self): + def pc(degree, vint): + return ootb._problem_class( + ootb.Facts("m", 3, set(), 10, vars_int=vint, model_degree=degree)) + self.assertEqual(pc("linear", 0), "LP") + self.assertEqual(pc("quadratic", 0), "QP") + self.assertEqual(pc("nonlinear", 0), "NLP") + self.assertEqual(pc("linear", 5), "MIP") + self.assertEqual(pc("quadratic", 5), "MIQP") + self.assertEqual(pc("nonlinear", 5), "MINLP") + # minus tier: not instantiated -> class unknown + self.assertIsNone(ootb._problem_class(ootb.Facts("m", 3, set(), 10))) + + +class TestSolverRoutingByClass(unittest.TestCase): + def setUp(self): + self.policy = ootb.load_policy() + + def _rec(self, degree, vars_int, available): + facts = ootb.Facts("m", 6, set(available), 10, effort="base", + vars_int=vars_int, vars_cont=5, model_degree=degree) + return ootb.recommend(facts, self.policy) + + def test_nonlinear_continuous_routes_to_ipopt(self): + d = self._rec("nonlinear", 0, {"gurobi", "ipopt"}) + self.assertEqual(d.problem_class, "NLP") + self.assertEqual(d.chosen_solver, "ipopt") # not gurobi + + def test_nonlinear_without_nlp_solver_picks_nothing(self): + # a MIP solver must NOT be chosen for a nonlinear model + d = self._rec("nonlinear", 0, {"gurobi", "cbc"}) + self.assertEqual(d.problem_class, "NLP") + self.assertIsNone(d.chosen_solver) + + def test_integer_model_never_routes_to_ipopt(self): + d = self._rec("linear", 5, {"ipopt", "cbc"}) + self.assertEqual(d.problem_class, "MIP") + self.assertEqual(d.chosen_solver, "cbc") # ipopt can't do integers + + def test_integer_nonlinear_is_minlp(self): + d = self._rec("nonlinear", 3, {"gurobi", "ipopt"}) + self.assertEqual(d.problem_class, "MINLP") + self.assertIsNone(d.chosen_solver) # no MINLP solver installed + + def test_user_solver_wins_over_class_routing(self): + facts = ootb.Facts("m", 6, {"gurobi", "ipopt"}, 10, effort="base", + vars_int=0, vars_cont=5, model_degree="nonlinear", + user_solver_name="cbc", + user_flags={"--solver-name"}) + d = ootb.recommend(facts, self.policy) + self.assertEqual(d.chosen_solver, "cbc") # OOTB defers to the user + + +if __name__ == "__main__": + unittest.main() diff --git a/mpisppy/utils/config.py b/mpisppy/utils/config.py index d49dc383b..e5945825f 100644 --- a/mpisppy/utils/config.py +++ b/mpisppy/utils/config.py @@ -1671,6 +1671,64 @@ def mmw_args(self): default=None, ) + def ootb_args(self): + """Out-of-the-box (OOTB) auto-configuration flags. + + Three mutually-exclusive effort tiers plus the (OOTB-independent) + ``--inspect-only`` dry run. See doc/designs/out_of_the_box_design.md + and mpisppy/generic/out_of_the_box.py. + + Each tier flag takes an OPTIONAL value: the path to a policy file. + Declared ``domain=str, default=None`` with ``nargs='?', const=''`` so + there are three states -- absent (None, OOTB off), bare flag ('', use + the shipped default policy), and ``--out-of-the-box PATH`` (that policy + file). A bool domain cannot be used because pyomo forces ``store_true`` + (which takes no value); ``str`` + ``nargs='?'`` is the supported + optional-value form. Detection keys on ``is not None`` (not truthiness) + because the bare-flag value is the empty string. + """ + self.add_to_config( + "out_of_the_box", + description="Auto-configure a defensible run from the environment " + "and model (base tier: one probe scenario). Optional value is a " + "policy-file path; bare flag uses the shipped default policy. " + "User-supplied options always win.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "out_of_the_box_minus", + description="Like --out-of-the-box but instantiates nothing " + "(structural decisions only; cannot bundle). Optional policy-file " + "path; bare flag uses the default policy.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "out_of_the_box_plus", + description="Like --out-of-the-box but instantiates all scenarios " + "and does a brief timed solve for more information (NOT a tuning " + "tool). Optional policy-file path; bare flag uses the default " + "policy.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": ""}, + ) + self.add_to_config( + "inspect_only", + description="Do the inspection (and, with --out-of-the-box-plus, " + "the brief calibration solve), print the configuration, the " + "equivalent command line, and config-time suggestions, then STOP " + "before the production run. Optional value is an assumed MPI rank " + "count for HPC planning (e.g. --inspect-only 512); bare flag uses " + "the actually-detected rank count.", + domain=str, + default=None, + argparse_args={"nargs": "?", "const": "detected"}, + ) + #================ def create_parser(self,progname=None): # seldom used diff --git a/run_coverage.bash b/run_coverage.bash index c89506f39..d5a81a0ae 100755 --- a/run_coverage.bash +++ b/run_coverage.bash @@ -122,6 +122,15 @@ run_phase "test_smps (serial)" \ run_phase "test_generic_cylinders (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_generic_cylinders.py -v +run_phase "test_out_of_the_box (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_out_of_the_box.py -v + +run_phase "test_ootb_validate (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_validate.py -v + +run_phase "test_ootb_calibrate (serial)" \ + coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_ootb_calibrate.py -v + run_phase "test_jensens (serial)" \ coverage run --rcfile=.coveragerc -m pytest mpisppy/tests/test_jensens.py -v