Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/developer_guide/testing_standards.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,35 @@ A score is 0 where its flag is False: an unusable number is never a failure. A s

**To add a shape:** if it is a number (a magnitude, a ratio, a start), add a field to `Case`, apply it in `values_of`, draw it in `cases` behind a `Domain` field, and add an example to `SEEDS`. If it is structure (an edge, a node size, a knob), add a topology to `search_topologies()` or a row to `KNOBS`; `_cells()` picks it up for the slow profile, and it joins `_FIRST` only if it should cost every push a compile (about 3 s). Then run the hunt over the new domain on three seeds (`search(name, cells=..., domain=..., profile=SLOW.seeded(k, shrink=False), fail=False)` gives the examples-to-find). An over-threshold example that is a defect goes into `KNOWN` with the claim it breaks; one that the claim does not promise goes into the score as an allowance with its reason. Prove a new score can fail with seeded faults in a scratch copy of `src/` (a halved factor, a dropped term, a flag forced True), each of which the per-push profile must find.

### A numerical reference, nonlinear cells, and twins with the same interface reading

The coupling search above scores against a closed form, which exists only for linear relays. Three instruments extend it to groups that have none: a nonlinear node today, and a mapping that reads a moving geometry once the diagnostics read one.

**The numerical reference** (`tests/property/coupling_reference.py`, `PassReference`) computes, in float64, what the closed form gave: the group's fixed point, the dense Jacobian of one pass at any iterate, and from them the spectral radius, the distance of a returned iterate in the norm the report states its bound in, the exact residual, and the error of the implicit gradient for every scalar constant.

- **How the pass map is obtained.** The caller builds an *x64 twin* of the graph under test: the same nodes, edges, schedule and sub-cycling in float64, its group at `max_iterations=1` with no acceleration and `predictor="linear"` (`twin_knobs`). One step of the twin is one pass, differentiated straight through. The pass starts from the predictor's extrapolation `2 pred_0 - pred_1`, which the step keeps in three `_meta` slots of the state; writing the iterate `x` into both gives `P(x; pre)`: one pass from the iterate `x`, each member integrating from the pre-step state. The map is the graph's own compiled step called on a state, so nothing of the estimator is called, and `jax.jacfwd` of it holds every dependence the pass has on the iterate (transforms, mappings, a geometry field a member holds). The slots' layout is found, not assumed: the twin is stepped once on a drawn example and the order of members whose flattened fields equal the stored `pred_0` is taken.
- **What it catches:** anything by which the estimator's numbers disagree with the map the solve iterates (a term of the pass missing from the estimator's Jacobian, a reading at another time level than the pass's, a Jacobian taken at another state than the returned one, a bound in another norm). **What it does not:** a defect in the pass itself, which the reference inherits (the closed-form and time-level references hold the pass); a difference between the single-pass branch and the pass an iterating group runs (checked separately: three passes of an iterating twin are three compositions of the reference's map, to float64 rounding); a node that branches on its dtype; a graph with more than one coupling group.
- **Validated against the closed form** on every per-push draw of the linear search (its 556 distinct examples on five cells, its seed shapes, and 150 draws on the multi-rate cell, in the slow lane; forty draws on one cell per push). Worst differences measured: the radius 1.0e-15; the Jacobian 1.6 float64 `eps` of its largest entry; the fixed point 5.7 `eps` of the field times the resolvent's norm; the distance 2.4e-12 of itself; the gradient error 1.9e-12. `ALLOWED` in `tests/property/test_coupling_nonlinear_search.py` holds each to at most 2**12 roundings.
- **Cost:** per twin, its build and two compiles (the pass with its Jacobian; the constants' sensitivities), together under two seconds for a dozen scalars; per example, a few dozen jitted calls (about 20 ms). The seconds of a cell are the compile of the graph under test with its diagnostics.

**The nonlinear cells** (`tests/property/test_coupling_nonlinear_search.py`) are structures of the linear search whose members pass each port's input through a nonlinearity centred on a parameter `c` before its gain: a saturating gain `c + tanh(s (u - c)) / s`, a quadratic term `u + s (u - c)^2`, or the product of two fields' deviations. Each has `phi(c) = c` and `phi'(c) = 1`, so with `c` the input at the linear group's fixed point the nonlinear group has the same fixed point and the same Jacobian *there*: the numbers of a case are drawn exactly as the linear search draws them, plus the curve `s max|c|` (0.01 to 100), and the linear closed form checks the reference's fixed point on every example. Away from the fixed point, where a capped solve returns, the Jacobian is another matrix, and only the reference knows it.

| Score | What changes for a map that is not affine |
|---|---|
| error bound (CPL-088) | the claim is for a linear `F` and "asymptotic" otherwise, so the distance is divided by the bound times `1 / (1 - h)`, with `h = ‖(I − J(x))⁻¹ (J_mean − J(x))‖` measured by the reference (`J_mean` the mean Jacobian between the fixed point and the returned iterate). `x − x* = (I − J_mean)⁻¹ (x − F(x))` exactly, so a bound that is right for the linearisation at the returned iterate is short by at most that factor: a score over one is a wrong number, not a nonlinear map. Unscored where `h ≥ 1` |
| spectral radius (CPL-087) | scored against the reference's Jacobian **at the returned iterate**, which is what the claim says and which differs from the fixed point's here |
| gradient bound (CPL-093, CPL-095) | two scores: over the gains, biases and mapping weights (`gradient`), and over each nonlinearity's `c` and `s` (`gradient_vanishing`), which the fixed point does not respond to, so that the relative error is of order one at any other iterate |
| floor (CPL-097, CPL-100) | the exact residual is the reference's |

Per push: one cell (`tri`, three members with one nonlinearity each, float32, five Jacobi passes), twenty derandomised examples a score, and two seed examples on which a number is within an eighth of its limit. Slow: 115 random examples a score on each of six blocks of seven cells (every structure, each nonlinearity, both dtypes, every configuration), not shrunk (a shrink is thousands of steps). An example that plain passes carry out of float range is not stepped on a cell with a quasi-Newton acceleration (see the pinned finding in the module). To add a nonlinearity, add a branch to `phi` with `phi(c) = c` and `phi'(c) = 1`; to add a structure, add it to `STRUCTURES` (single-rate: a sub-cycled member reads its input interpolated, and `c` would not be its input at the fixed point).

**The twins with the same interface reading** (`tests/property/geometry_graphs.py`, `tests/property/test_interface_reading_twins.py`). `convergence_norm="interface"` reads what each internal edge delivers. The node-inlined twin of a geometry edge moves the mapping into the *target*, so its edges deliver the raw source field and its interface norm is another norm. Two twins keep the reading:

- `transform_twin` writes a static mapping as its edge's transform: the same state and the same reading, so every number of the report but the gradient bound is the edge-mapped graph's (bit for bit as measured; held to 2**10 `eps`). The gradient bound probes the step's parameters, and the weights are constants of a transform.
- `relay_twin` moves the mapping into a relay fused with the *source* node, whose state field is the mapped value; a plain edge delivers it. The reading is the same entry for entry, and a geometry the mapping reads sits inside the relay (source anchors only: a target-anchored geometry would reach the relay over an internal edge the norm would read too, and so would a relay that is a node of its own). The state has the relay's field in it, so what is equal is what the reading decides: every iterate of an unaccelerated solve, the residual, the pass count, the verdict, `rho_spectral` where the spectrum is resolved, the flags. `spectral_error_bound`'s factor is 1.1% to 3.4% apart (it is the norm of a resolvent compressed onto a Krylov basis of the state), the float floor 13% apart, and the gradient bound is in another norm; under Jacobi the relay doubles the scalars a pass carries, so the comparison is made under Gauss-Seidel.

On a static mapped edge both twins report as the edge-mapped graph does, and a fault seeded in the library's reading (the edge read without its mapping) puts the two residuals 8.8% apart. On the solve path the relay twin of a *moving* source-anchored geometry steps as the edge-mapped graph bit for bit, and a geometry read from the pre-step state in place of the iterate breaks that after two passes (a converged solve does not see it: at a fixed point the two agree). The interface norm over a geometry edge is refused in 0.4.0; `RELAY_INTERFACE_CASES`, in the harness's phase-1 block, are the cases whose reports are compared with their relay twins' once `DIAGNOSTICS_READ_GEOMETRY` is set. The harness module's docstring lists the six faults that later work is held to, the instrument for each and the signal measured where the fault can be seeded today.

## Test time budget

Every push runs the default lane (everything not marked
Expand Down
4 changes: 4 additions & 0 deletions tests/duration_allowlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,7 @@ tests/property/test_sysid_targeted_search.py::test_no_wrong_fit_among_the_fixed_
tests/compliance/test_claims_inventories.py::test_every_cited_test_exists_and_runs_on_every_push[coupling_claims.yaml] # kept: the guard that every test a coupling claim cites exists and is collected per push; it must run on every push, and its time is a subprocess collection that grows with the number of cited tests (20.3 s and 20.8 s on CI)
tests/property/test_coupling_targeted_search.py::test_a_usable_error_bound_is_never_below_the_distance_per_push # kept: the per-push witness of the slow hunt over coupling_diagnostics()'s four numbers; its seconds are the four compiled cells (solver="ift" with diagnostics) that the other three searches, the seeds and the pins then share
tests/core/test_coupling_claims_in_every_domain.py::test_the_claim_holds_for_each_member_of_a_vmapped_step[CPL-006] # kept: the per-push witness of the vmap cell of CPL-006; its time is tracing a batched coupled step with diagnostics on, and it measured 20.2 s and 21.9 s on CI against the 20 s line (2026-10-06), the second with the ninth Jacobian-vector product of the spectral self-check
tests/property/test_coupling_nonlinear_search.py::test_the_reference_reproduces_the_closed_form_on_a_linear_cell # kept: the per-push check that the numerical coupling reference is the closed form (the reference is what the nonlinear cells and the later geometry cells are scored against); its seconds are one cell of the linear search compiled (which that search then does not pay) and its twin
tests/property/test_coupling_nonlinear_search.py::test_the_nonlinear_fixed_point_is_the_linear_one_and_the_jacobian_moves # kept: carries the compile of the one per-push nonlinear cell (solver="ift" with diagnostics) and of its float64 twin, which the four per-push nonlinear searches then share; the coupling search has no other nonlinear coverage per push
tests/property/test_interface_reading_twins.py::test_a_static_mapped_edge_under_the_interface_norm_reports_a_usable_spectrum[float32 {'max_iterations': 3}] # kept: carries the compile of the edge-mapped graph the per-push interface-reading comparison is made against (a group with diagnostics under the interface norm)
tests/property/test_interface_reading_twins.py::test_a_static_mapped_edge_under_the_interface_norm_reports_as_its_transform_twin[float32 {'max_iterations': 3}] # kept: the only per-push comparison of coupling_diagnostics() under the interface norm with a twin that has the same reading (a reading without its mapping moves the residual by 8.8% here and nothing else per push sees it); its seconds are the twin's compile
Loading
Loading