fix(core): a step that changes the layout of the state is refused (MADD-ANO-220)

CHANGELOG.md

@@ -215,6 +215,7 @@ guidance; the itemized changes follow.
   (stateful machines), the params pytree, `sysid`, retracing and binary frames
 
 ### Changed
+- **A step that changes the layout of the state raises `ValueError`; it used to be stored** (released behaviour, corrected: `MADD-ANO-220`): `GraphManager.step`, `run`, `run_adaptive` and `FmuSidecar.step` refuse a stepped state whose leaf has another shape or kind of dtype than the state it replaces, or whose fields differ, naming the node and the leaf; nothing is stored. A node's `update` must return the layout `initial_state()` built: build a state that grew at its first step at its final shape.
 - **`add_coupling_group` refuses a boolean option that is not a boolean, and `rtol=0` under the norms that divide by it**: `diagnostics`, `subcycling` and `strict_convergence` raise `TypeError` for anything but `True`/`False` (`diagnostics="off"` turned the diagnostics on); `rtol=0` under `convergence_norm="mixed"` or `"interface"` raises `ValueError` (the residual was `0/0`, the step ran to its cap and the report raised `ZeroDivisionError`). Action: pass a bool; use a positive `rtol`.
 - **A multi-rate graph whose schedule would not keep a node's clock no longer compiles** (MADD-ANO-227, in every release to 0.3.1): timesteps more than about 1e9 apart (`1.0` and `1e-10` got rate dividers of 1 and 0 and ran on different clocks, silently), or with no common step the float GCD finds, are a `ValueError` at `compile()` naming the two nodes, an error from `validate()`, and a 400 from `POST /graph/nodes` and `DELETE /graph/nodes/{name}`. The dividers of every graph that is kept are unchanged. Give the nodes timesteps that are whole multiples of one step.
 - **REST numbers are strict, and counts are held to their declared range** (MADD-ANO-228): `"timestep": true` (a node at 1.0 s), `"0.5"` and `" 0.25 "` are a 422, as are a boolean or text for any number of a request model; an integer query parameter is decimal digits only; `POST /sim/profile` answers 422 for `n_steps` outside 1..1000 or `n_warmup` outside 0..50 where it clamped them without saying so. Send JSON numbers, and counts inside the documented range.
@@ -417,14 +418,15 @@ guidance; the itemized changes follow.
   The `[verify]` extra now only pulls `hypothesis`.
 
 ### Fixed
+- **`GraphManager.step`, `run` and `run_adaptive` no longer store a step that reshapes a state leaf** (`MADD-ANO-220`, in 0.1.0 to 0.3.1): a list given for a scalar constant (`BallNode(initial_velocity=[1.0, 2.0])`), or an external input of another shape at a later step, broadcast a scalar leaf, and the state's checkpoint then did not load after `reset_state()`. The step is refused by name (compared once per trace, host-side: no compiled program changes), as `run_scan` always did; the FMU sidecar and bridge refuse it too.
 - **`coupling_diagnostics()`: three numbers that read wrong with their flag set, and one found beside them** (MADD-ANO-222, 225, 226, 229, never released): `gradient_relative_error_bound` at the float floor took the change along one stand-in direction (5.1e-6 for 3.0e-5); `rho_spectral` read settled past eight interface scalars (0.273 for 0.219), from sampled rounding on a float32 group far from normal (0.250 for 0.206), and lost a loop below a field's rounding on a sub-cycled group (1e-12 for 1.25e-4).
   The gradient bound's undirected distance now takes an operator norm; a Krylov space still growing at the cap is never settled (`spectral_usable=False` above seven interface scalars, eight where they are the whole state); a certificate over every perturbation of the measured size replaces the samples; a sub-cycled boundary value is differentiated as `(1 - alpha) a + alpha b` (values unchanged, gradients of sub-cycled groups move at rounding level).
   Action: none; expect `spectral_usable=False` on more float32 and 16-bit groups whose Jacobian is far from normal (usable fraction on the search's draws 0.979 to 0.976).
 - **`POST /checkpoint/load` takes a checkpoint saved before an initial condition was written** (never released): after `PUT /graph/params {initial_*: v}` (or `TableNode.position`) every earlier checkpoint was a 400 "does not fit this graph", for six of the sixteen numeric parameters of the stock nodes. The load now installs each leaf it changes as `PUT` does (in the node too, so the next reset builds the state from it), answers as `PUT` of the same value on every parameter of every built-in node, and its 400 names the checkpoint's value and the route that fixes it.
 - **`run_pod.py --summarise` reads a goal file with no cells or no checks INVALID (exit 3) for every goal** (never released): an emptied `forward.json`, `gradient.json` or `exchange.json` read "no checks" and the summary exited 0.
 - **A constructor's refusal at `POST /graph/nodes` names the parameter** it can be told from, the value sent and the class's default (`n_cells: 8.0` was "'float' object cannot be interpreted as an integer").
 - **An FMU parameter whose `ParamSpec` accepts no value advertises an empty `min` / `max`** (never released; the spec-level refusals are 0.4.0's own): a `logit` spec with a subnormal float32 bound, or a `log` / `logit` bound or width the leaf's dtype does not hold, is refused by `check_params`, the sidecar and REST for every value, while `build_model_description` advertised the neighbours of its bounds (`min = 1.18e-38`, `max` just under `1.42e-14` for `logit` on `(1.4e-45, 1.42e-14)`), so a bridge over a sidecar built without `param_specs` took every value between them. The variable now advertises `min = tiny > max = -tiny`, and the bridge refuses to serve the description, with or without specs. Action: none; narrow the bounds as the refusal says.
-- **`POST /graph/nodes` accepts only a node the graph can step with its state as built** (the REST door of `MADD-ANO-220`, which stays open in process: `GraphManager.step` stores a step that changes a leaf's shape; use `run_scan`, which refuses it): one update is traced as the graph calls it (params pytree, with and without boundary inputs) and held to `initial_state()`'s fields, shapes and kinds of dtype; `venous_pressure: null` and a list for a scalar constant are a 400 naming the parameter. Also: lists and objects count against the 10^6 params bound; `POST /checkpoint/load` names why a graph cannot compile; `POST /surrogate/deactivate` (experimental) lists the edges it drops (`dropped_edges`).
+- **`POST /graph/nodes` accepts only a node the graph can step with its state as built** (the REST door of `MADD-ANO-220`; in process the step itself refuses such a node, see the entry above): one update is traced as the graph calls it (params pytree, with and without boundary inputs) and held to `initial_state()`'s fields, shapes and kinds of dtype; `venous_pressure: null` and a list for a scalar constant are a 400 naming the parameter. Also: lists and objects count against the 10^6 params bound; `POST /checkpoint/load` names why a graph cannot compile; `POST /surrogate/deactivate` (experimental) lists the edges it drops (`dropped_edges`).
 - **`run_pod.py` refuses an option value no goal can use with exit 2, before the backend loads** (never released): `--cells 0`, `--warmup -1`, `--repeats 0`, `--steps 0`, a `--mesh` that cannot be read. They used to pass, or exit 5 as a crashed goal.
 - **`rho_spectral` and `spectral_usable` under `diagnostics=True`** (MADD-ANO-221, 223, 224, never released): the spectral estimate took a Krylov direction below 1e-5 of a product for rounding in every dtype, took the compressed radius by repeated squaring, and called a spectrum settled on the Arnoldi residual alone; `rho_spectral` read 0.379 for 0.5 (float64), 0.0512 for 0.0500 (float32) and 0.941 for 0.735 (twelve non-normal scalars), and `spectral_error_bound` 0.61x the distance, each with `spectral_usable=True`.
   The breakdown test is now eight units of the products' own rounding, the radius is `eigvals`, and one more Jacobian-vector product (nine per group per step, was eight) checks the estimate against itself: where it moves the radius by more than 5% of `1 - rho_spectral` the flag is `False`.

docs/developer_guide/node_authoring.md

@@ -9,6 +9,7 @@ A MADDENING {term}`node <Node>` is a **{term}`pure function <Pure function>` wra
 - `initial_state(params) -> dict` — returns the initial state arrays
 - `update(state, boundary_inputs, dt) -> new_state` — returns a new state dict
 - `update()` must be **{term}`JAX-traceable`**: no Python-level side effects, no data-dependent control flow, no print statements. Use `jnp.where` instead of `if/else`.
+- `update()` returns the state layout it was given: the same fields, each with the shape and kind of dtype `initial_state()` built. A step that changes one (a list passed for a scalar constant broadcasts a scalar field) is refused by `GraphManager.step`, `run` and `run_adaptive` with a `ValueError` naming the node and the field, as `run_scan` refuses it.
 - State is **immutable** — return a new dict, don't mutate in place
 - Parameters live in `self.params`, not in state
 

docs/developer_guide/sharding_topology.md

@@ -253,12 +253,14 @@ What 0.4.0 added:
   integral can leave out the padding of a short shard's block;
 * refusals of a node declaring a Cartesian `halo_width()` and of a node
   whose cell count is not the layout's (both silently wrong before);
-* a domain integral carried in the state (a graph's second step, or an
-  initial value the node declares) is placed as the step returns it —
-  replicated once reduced, stacked along the mesh axis otherwise — rather
-  than partitioned, so such a node runs in a `GraphManager` (the stencil
-  wrapper had the same defect for a vector or per-shard integral, fixed the
-  same way);
+* a domain integral carried in the state (an initial value the node
+  declares, or a step's output handed back to the wrapper) is placed as
+  the step returns it — replicated once reduced, stacked along the mesh
+  axis otherwise — rather than partitioned, so such a node runs in a
+  `GraphManager` (the stencil wrapper had the same defect for a vector or
+  per-shard integral, fixed the same way).  In a graph the node gives the
+  integral an initial value in `initial_state()`: a step that adds a field
+  to the state is refused (`GraphManager.step`, as `run_scan`);
 * the session runner `benchmarks/multigpu/run_pod.py` (and its CPU
   `--dry-run`).
 

docs/release_notes/v0.4.0.md

@@ -7148,8 +7148,7 @@ fixed](#the-known-coupling-sysid-and-fmu-findings-fixed)".
 
 - **MADD-ANO-217** (**open**; carried by no release before this one): `fit_lm` from a stiffness started 18 to 80 times too high can stop short, `converged=False`, on the end of a wide damping range; start it again from the returned parameters.
 
-`MADD-ANO-220` — *GraphManager.step, run and run_adaptive store a step that
-changed the shape of a state leaf* — **open**.
+- **MADD-ANO-220** (resolved; shipped in every tagged release): `step`, `run` and `run_adaptive` stored a step that changed the shape of a state leaf (a list given for a scalar constant), after which the state's checkpoint did not reload; such a step now raises `ValueError` and stores nothing.
 
 `MADD-ANO-230` — *A direction the Arnoldi breakdown test takes for rounding can
 carry the dominant mode of a Jacobian far from normal: float32 fan-out hubs

docs/validation/known_anomalies.yaml

@@ -4287,7 +4287,7 @@ anomalies:
       - "tests/cloud/multigpu/test_integral_listed_as_state_field.py::test_an_integral_listed_in_state_fields_is_summed_over_the_shards"
       - "tests/cloud/multigpu/test_unstructured_layout_contract.py::test_a_domain_integral_fed_back_as_state_steps_again"
       - "tests/cloud/multigpu/test_unstructured_layout_contract.py::test_a_node_declaring_its_integral_runs_in_a_graph_like_the_unsharded_node"
-      - "tests/cloud/multigpu/test_unstructured_layout_contract.py::test_a_node_without_an_initial_integral_steps_in_a_graph"
+      - "tests/cloud/multigpu/test_unstructured_layout_contract.py::test_a_node_without_an_initial_integral_is_handed_its_own_output_back"
       - "tests/cloud/multigpu/test_stencil_domain_integral_state.py::test_a_vector_domain_integral_in_the_state_runs_in_a_graph_like_the_unsharded_node"
       - "tests/cloud/multigpu/test_stencil_domain_integral_state.py::test_a_per_shard_domain_integral_fed_back_by_a_graph_steps_again"
       - "tests/cloud/multigpu/test_stencil_domain_integral_state.py::test_the_initial_value_of_an_integral_is_placed_as_the_step_returns_it"
@@ -14929,12 +14929,23 @@ anomalies:
       cannot use (`HeartPumpNode` `venous_pressure: null`), after which
       every `POST /sim/step` answered 400, is refused there too.
 
-      In process the step entry points are unchanged.  A check of the
-      stepped state's keys and shapes in `step` (once per compile,
-      host-side) was written and withdrawn: it refuses the fixture the
-      compile-count gate's own tests use to provoke a retrace
-      (`tests/core/test_compile_counts.py`, a state that grows on its
-      first step), so it waits for a decision on that fixture.
+      Resolved in 0.4.0 in process too.  `step` and `run` compare the
+      stepped state with the state it replaces -- keys, shapes and kinds
+      of dtype (boolean, integer, float, complex) -- once per trace of the
+      compiled step, host-side, and `run_adaptive` compares the first
+      step it keeps; a difference is a `ValueError` naming the node and
+      the leaf, and nothing is stored (the graph holds the state it held,
+      no observer is told of a step, no callback is called).  A step that
+      is traced again (a new compile; an external input of another shape,
+      which broadcast a scalar leaf at a later step) is compared again.
+      A leaf whose kind of dtype changed (an integer count returned as a
+      float) was stored the same way and is refused the same way.  The
+      FMU sidecar and bridge, which run the compiled step on a state of
+      their own, make the same comparison (`FmuSidecar.step`; an error
+      reply from the bridge, nothing committed).  Nothing is traced and
+      no node's `update` is called for it: the compiled programs are
+      unchanged.  A dtype's width is not compared (with x64 enabled a
+      stock node returns float64 for a float32 state).
     severity: "major"
     safety_relevance: "context_dependent"
     safety_relevance_rationale: >
@@ -14947,22 +14958,39 @@ anomalies:
       - "maddening.core.graph_manager.GraphManager.step"
       - "maddening.core.graph_manager.GraphManager.run"
       - "maddening.core.graph_manager.GraphManager.run_adaptive"
-    affected_versions: ">=0.1.0"
+      - "maddening.fmi.sidecar.FmuSidecar.step"
+    affected_versions: ">=0.1.0, <0.4.0"
     workaround: >
-      Give a node's constants the rank its state has (a scalar for a scalar
-      field), and advance a graph you have not checked with `run_scan`,
-      which refuses a step that changes the state's layout.  Over REST,
-      0.4.0 refuses such a node where it is added.
-    resolution_status: "open"
-    resolution_version: null
-    residual_risk: null
+      On 0.1.0 to 0.3.1 give a node's constants the rank its state has (a
+      scalar for a scalar field), and advance a graph you have not checked
+      with `run_scan`, which refuses a step that changes the state's
+      layout.
+    resolution_status: "resolved"
+    resolution_version: "0.4.0"
+    residual_risk: >
+      Changed behaviour: a step that changes the shape or the kind of
+      dtype of a state leaf, or the fields of a node's state, raises
+      `ValueError` where it was stored; a graph that relied on its state
+      growing at the first step must build the state at its final layout
+      in `initial_state()`.  That includes a node whose `update` returns a
+      field `initial_state()` does not build (a domain integral it only
+      emits): `step` stored the first step of such a node, sharded or
+      not, and now refuses it; the node gives the field an initial value.
+      A dtype of another width of the same kind is still stored as
+      returned.
     verification:
-      # A strict xfail: a change that makes the step entry points keep or
-      # refuse the layout is noticed.
       - "tests/core/test_a_step_keeps_the_state_layout.py::test_the_step_entry_points_keep_the_shape_of_every_state_leaf"
+      - "tests/core/test_a_step_keeps_the_state_layout.py::test_a_step_that_changes_a_leafs_kind_of_dtype_or_the_fields_is_refused"
+      - "tests/core/test_a_step_keeps_the_state_layout.py::test_a_later_step_that_is_traced_again_is_compared_again"
+      - "tests/core/test_a_step_keeps_the_state_layout.py::test_the_step_of_a_graph_compiled_again_is_compared_again"
+      - "tests/core/test_a_step_keeps_the_state_layout.py::test_the_comparison_is_made_once_per_trace_and_calls_no_update"
       - "tests/core/test_a_step_keeps_the_state_layout.py::test_the_scan_entry_points_refuse_a_step_that_changes_a_leafs_shape"
+      - "tests/fmi/test_a_sidecar_step_keeps_the_state_layout.py::test_a_sidecar_step_that_reshapes_a_leaf_is_refused_and_nothing_is_committed"
+      - "tests/cloud/multigpu/test_a_sharded_step_keeps_the_state_layout.py::test_a_node_beside_a_sharded_one_that_reshapes_its_state_is_refused"
       - "tests/api/test_a_new_node_is_one_the_graph_can_step.py::test_a_value_of_a_rank_that_reshapes_the_state_at_the_first_step_is_a_400"
+      - "tests/api/test_a_new_node_is_one_the_graph_can_step.py::test_a_node_the_door_never_saw_is_refused_where_the_graph_steps"
       - "tests/property/test_rest_requests_generated_from_the_schema.py::test_a_new_node_of_any_kind_is_refused_or_steps_with_its_layout_and_reloads"
+
   - anomaly_id: "MADD-ANO-221"
     title: "The Arnoldi breakdown test was 1e-5 of the product in every dtype: rho_spectral read 0.379 for 0.5 and spectral_error_bound 0.61x the true distance, spectral_usable=True"
     description: >

docs/validation/rest_runpod_claims.yaml

@@ -2295,14 +2295,18 @@ claims:
     a configuration error found when the step is traced (an edge of mismatched
     shapes, constraints the step cannot use), and a run-time check in the step
     (equinox.error_if) raising at the first step, or in the sixth doubling
-    slice of a run after 32 steps were stored; a step of an edited graph
+    slice of a run after 32 steps were stored; a step that changes the layout
+    of the state (a node the graph was given in process whose scalar leaf
+    broadcasts, which the step stored before 0.4.0: MADD-ANO-220), refused by
+    the graph on /sim/step and /sim/run; a step of an edited graph
     whose compile refuses; a failure injected at every point of a step's
     body. Not claimed for simultaneous
     requests.
   oracle: 400 naming the error; steps_run against the graph's state and the relay's count.
   tests:
     - tests/api/test_params_writes_the_step_cannot_run_with.py::test_a_graph_configured_so_it_cannot_step_is_a_400_naming_why
     - tests/api/test_runner_pacing_and_status.py::test_an_edge_the_graph_cannot_validate_is_a_400_on_every_route_that_steps
+    - tests/api/test_a_new_node_is_one_the_graph_can_step.py::test_a_node_the_door_never_saw_is_refused_where_the_graph_steps
     - tests/api/test_routes_answer_what_they_did.py::test_a_single_step_that_raises_is_a_400_and_stores_nothing
     - tests/api/test_routes_answer_what_they_did.py::test_a_run_that_raises_at_its_first_step_is_a_400_and_moves_nothing
     - tests/api/test_routes_answer_what_they_did.py::test_a_run_that_raises_part_way_says_how_many_steps_it_took