Graph-layer architecture review: model/renderer split, correctness + perf fixes, and issue #1 null-flow routing switch

CHANGELOG.md

@@ -1,5 +1,81 @@
 # Changelog
 
+## [0.3.3] - 2026-07-16
+
+Large architecture-review pass over the `alphaDeesp` graph layer: correctness
+fixes, performance work, two deep revisions, the interactive-viewer
+decomposition, and the fix for the multigraph Dijkstra-weight bug (issue #1).
+All changes are unit-tested (372 runnable tests without grid2op) and checked by
+adversarial multi-agent verification. Public import paths are unchanged.
+
+### Bug Fixes
+
+- **Multigraph Dijkstra weight in the null-flow path search** (issue #1): on the
+  overflow `MultiDiGraph`, networkx passes a callable weight the `{key: attr}`
+  parallel-edge view, so `_compute_sssp_paths`'s `attr.get("capacity", 0)`
+  silently read `0` and routing was hop-cost-only. The routing weight is now
+  precomputed as an edge attribute (string-weight Dijkstra) with a
+  `capacity_weighted` switch: `False` (default) reproduces the historical
+  behaviour **bit-identical** ("bless"); `True` enables capacity-weighted routing
+  with correct min-parallel capacity and `(u,v)`/`(u,v,key)` promoted matching
+  ("fix"). Threaded through `add_relevant_null_flow_lines[_all_paths]` so
+  downstream callers can opt in without patching.
+- **`shortest_paths.py`**: removed a dead branch and made the promoted-edge
+  matching multigraph-correct (was silently weighting every parallel edge as `0`).
+- **`OverFlowGraph.__init__` no longer mutates the caller's DataFrame** (copies
+  it); `rename_nodes` loop variable clarified (behaviour unchanged).
+- **`Structured_Overload_Distribution_Graph.get_dispatch_edges_nodes`** guards
+  the empty-loops case (`red_loops.Path.sum()` no longer raises `TypeError` on a
+  grid with no loop paths).
+- **`to_DiGraph`** defaults a missing edge `capacity` to `0.0` (was `1.0`), which
+  no longer skews `rank_red_loops`' min-cut.
+
+### New Features / Deep revisions
+
+- **`OverFlowGraph` model/renderer split**: new
+  `graphs/overflow_renderer.py::OverflowGraphRenderer` owns all Graphviz
+  presentation (penwidth, shapes, tapered styling, compound highlight colours,
+  plotting); `OverFlowGraph` keeps the semantic model. New
+  `graphs/edge_roles.py::edge_role_of` (+ `EDGE_ROLE_*`, `OverFlowGraph.edge_role`)
+  is the single authority mapping an edge's base colour to a semantic role, so no
+  consumer parses Graphviz colour strings; `highlight_significant_line_loading`
+  records an authoritative `base_color`.
+- **`AlphaDeesp` explicit pipeline**: the ranking pipeline moved to `run()`;
+  `AlphaDeesp(..., auto_run=True)` (default) preserves the previous behaviour,
+  `auto_run=False` builds the object without side effects.
+- **Lazy `Structured_Overload_Distribution_Graph`**: colour-filtered views,
+  `red_loops` and `hubs` are `functools.cached_property` computed from a
+  construction-time snapshot — behaviour-identical and order-independent.
+
+### Performance
+
+- `delete_color_edges` accepts a colour **or an iterable of colours**, removing
+  them in a single graph copy; the structured graph builds its derived views in
+  single passes.
+- `sort_hubs` and the initial-inflow lookup in `AlphaDeesp` are vectorised
+  (were per-hub / per-edge `iterrows` scans).
+- `simulation.create_df` is vectorised (was several `iterrows` passes;
+  400-case differential fuzz pins equivalence).
+- Null-flow Dijkstra uses a precomputed string weight instead of a per-edge
+  Python callable.
+
+### Maintainability
+
+- `core/interactive_html.py` (976 LOC) split into the `core/interactive_html/`
+  package (8 focused modules) with the CSS/JS/HTML skeleton externalised under
+  `assets/` and reassembled byte-exactly at runtime (`package_data` shipped).
+- `find_loops` / consolidation path enumeration gained opt-in cutoffs
+  (`loop_path_cutoff`, default `None` = unbounded = original behaviour).
+- New `docs/CODE_REVIEW.md` (full review + downstream-impact analysis on
+  `Expert_op4grid_recommender` + remaining-work backlog); `CLAUDE.md` refreshed.
+
+### Tests
+
+- New suites: `test_edge_roles.py`, `test_overflow_renderer.py`,
+  `test_simulation_create_df.py`, `test_null_flow_weighting.py`; expanded
+  `test_graphs_package.py`, `test_overflow_graph.py`, `test_alphadeesp_unit.py`,
+  `test_shortest_paths.py`, `test_interactive_html.py`.
+
 ## [0.3.2.post4] - 2026-06-17
 
 ### New Features

CLAUDE.md

@@ -24,18 +24,38 @@ alphaDeesp/
 ├── Expert_rule_action_verification.py  # Rule-checking utilities for proposed actions
 ├── core/
 │   ├── alphadeesp.py               # AlphaDeesp algorithm (ranking, topology exploration)
-│   ├── graphsAndPaths.py           # OverFlowGraph, PowerFlowGraph, Structured_Overload_Distribution_Graph
+│   ├── topology_scorer.py          # TopologyScorerMixin (score helpers for AlphaDeesp)
+│   ├── topo_applicator.py          # TopoApplicatorMixin (apply topo vectors to the graph)
+│   ├── twin_nodes.py               # Twin-node id scheme (busbar-split node encoding)
+│   ├── graphsAndPaths.py           # Back-compat shim → re-exports from core/graphs/
+│   ├── graphs/                     # Graph layer (split out of the old graphsAndPaths.py)
+│   │   ├── overflow_graph.py       # OverFlowGraph — the overflow *semantic model*
+│   │   ├── overflow_renderer.py    # OverflowGraphRenderer — Graphviz *presentation* only
+│   │   ├── power_flow_graph.py     # PowerFlowGraph (base current-state graph)
+│   │   ├── structured_overload_graph.py  # Structured_Overload_Distribution_Graph
+│   │   ├── constrained_path.py     # ConstrainedPath value object
+│   │   ├── null_flow_graph.py      # NullFlowGraphMixin (null-flow line handling)
+│   │   ├── null_flow.py            # double/un-double null-flow edge helpers
+│   │   ├── graph_consolidation.py  # GraphConsolidationMixin (disambiguation)
+│   │   ├── graph_utils.py          # pure networkx helpers (delete_color_edges, ...)
+│   │   ├── shortest_paths.py       # mandatory/promoted-edge shortest paths
+│   │   ├── edge_roles.py           # base-colour → semantic role accessor (edge_role_of)
+│   │   └── constants.py            # default_voltage_colors, ...
+│   ├── interactive_html/           # Interactive HTML/SVG overflow viewer (used by other repos)
+│   │   ├── {helpers,constants,layers,model,svg,template,render}.py
+│   │   └── assets/{viewer.css,viewer.js,template.html}  # externalised JS/CSS/skeleton
 │   ├── simulation.py               # Abstract Simulation base class + DataFrame plumbing
 │   ├── network.py                  # Network / Substation model objects
-│   ├── elements.py                 # Production, Consumption, OriginLine, ExtremityLine dataclasses
+│   ├── elements.py                 # Production, Consumption, OriginLine, ExtremityLine classes
 │   ├── printer.py                  # Graphviz/shell printing helpers
 │   ├── grid2op/                    # Grid2op backend (Grid2opSimulation, Grid2opObservationLoader)
-│   └── pypownet/                   # Pypownet backend (legacy / optional)
+│   └── pypownet/                   # Pypownet backend (legacy / deprecated)
 ├── ressources/
 │   ├── config/config.ini           # Default runtime config (thresholds, layout, simulator type)
 │   └── parameters/                 # Built-in grids (l2rpn_2019, rte_case14_realistic, custom14, ...)
 └── tests/                          # pytest suite (unit + integration, grid2op + pypownet)
-docs/                               # Sphinx sources (RST)
+docs/                               # Sphinx sources (RST) + CODE_REVIEW.md
+scripts/code_quality_report.py      # Aggregated radon/vulture/ruff/interrogate report
 getting_started/                    # Jupyter tutorial notebooks
 .circleci/config.yml                # CI: pytest on cimg/python:3.12 with graphviz
 ```
@@ -57,7 +77,7 @@ getting_started/                    # Jupyter tutorial notebooks
               │ expert_operator.expert_operator()  │
               └────────┬───────────────────────────┘
                        │
-                       ├─► OverFlowGraph (graphsAndPaths.py)
+                       ├─► OverFlowGraph (core/graphs/overflow_graph.py)
                        ├─► AlphaDeesp.get_ranked_combinations()
                        └─► sim.compute_new_network_changes() -> end-result DataFrame
 ```
@@ -66,6 +86,56 @@ Key contract: any new simulator backend must implement the abstract methods of
 `alphaDeesp/core/simulation.py::Simulation` (get_dataframe, isAntenna,
 get_substation_elements, compute_new_network_changes, etc.).
 
+### The `graphs/` package (and `OverFlowGraph`)
+
+`core/graphsAndPaths.py` is now a **backwards-compatible shim** that re-exports
+the public surface of the `core/graphs/` package. External code (and other
+repos) can keep importing `from alphaDeesp.core.graphsAndPaths import
+OverFlowGraph, ...`; new code should import from `alphaDeesp.core.graphs`.
+
+`OverFlowGraph` is split into a **semantic model** and a **renderer**:
+
+- `graphs/overflow_graph.py::OverFlowGraph` owns the semantic model — graph
+  topology, per-edge redispatch magnitude, edge *role* encoded as a base
+  colour (`black` overload / `blue` negative / `coral` positive / `gray`
+  insignificant), and the boolean semantic flags consumed downstream
+  (`is_overload`, `is_monitored`, `on_constrained_path`, `in_red_loop`,
+  `is_hub`, `is_extra_cut`). The public method signatures are unchanged.
+- `graphs/overflow_renderer.py::OverflowGraphRenderer` owns all Graphviz
+  *presentation* (penwidth scaling, node shapes, tapered swap styling, the
+  compound `"colour:yellow:colour"` highlight strings and HTML loading
+  labels, and plotting). It is **stateless** (static methods over a passed-in
+  graph) so downstream repos can reuse it on any compatible `MultiDiGraph`.
+
+The public surface of the package is pinned by `tests/test_graphs_package.py`
+(`EXPECTED_PUBLIC_NAMES` / `EXPECTED_SUBMODULES`) — update those sets when you
+add or move a public symbol.
+
+**Semantic edge roles.** `graphs/edge_roles.py::edge_role_of(edge_data)` is the
+single authority mapping an edge's base colour to a stable role
+(`EDGE_ROLE_OVERLOAD/NEGATIVE/POSITIVE/INSIGNIFICANT/NULL_NON_RECONNECTABLE`).
+It prefers the `base_color` attribute (recorded by the renderer when it wraps a
+colour into a compound `"c:yellow:c"` highlight) and is compound-safe — so no
+consumer should ever parse a Graphviz colour string. `OverFlowGraph.edge_role(name)`
+is the convenience by-line-name accessor.
+
+**Lazy structured graph.** `Structured_Overload_Distribution_Graph` computes its
+colour-filtered views, `red_loops` and `hubs` as `functools.cached_property`
+(constrained path stays eager). `red_loops` uses the constructor *seed* hubs;
+the public `find_loops()` re-enumerates with the *detected* hubs (this split is
+what makes the lazy properties order-independent while matching the old eager
+behaviour). Don't reintroduce eager computation.
+
+**AlphaDeesp construction.** `AlphaDeesp(..., auto_run=True)` runs the ranking
+pipeline in the constructor (default, backwards-compatible). Pass
+`auto_run=False` and call `.run()` for staged/testable execution.
+`AlphaDeesp_warmStart` is the pre-existing "skip the pipeline" path.
+
+**Interactive viewer.** `core/interactive_html/` is a package; the CSS/JS/HTML
+skeleton are externalised under `assets/` and reassembled at runtime by
+`template.html_template()`. Edit the `.css`/`.js` assets directly. The package
+is shipped via `package_data` in `setup.py` + `MANIFEST.in`.
+
 ## Common Commands
 
 ```bash
@@ -120,7 +190,12 @@ Runtime: `Grid2Op`, `lightsim2grid`, `networkx`, `rustworkx`, `pandapower`,
 
 Optional: `pypownet>=2.2.0`, `oct2py`, `pypower` (for the legacy backend).
 
-Python: targeted at 3.12 in CI; `setup.py` still advertises 3.6/3.7 (stale).
+Python: `setup.py` declares `python_requires=">=3.9"` with classifiers for
+3.9–3.12; CI runs on 3.12. Note the `graphs/` package unit tests
+(`test_overflow_graph.py`, `test_graphs_package.py`, `test_null_flow.py`, ...)
+run **without** grid2op; the grid2op integration suites, `alphadeesp_test.py`,
+`test_cli.py` and `test_expert_rules.py` require `grid2op` + `lightsim2grid`
+installed.
 
 Graphviz **executables** must be on PATH for snapshot/plot mode (not just the
 Python binding). On Debian/Ubuntu: `apt-get install graphviz`.
@@ -134,14 +209,33 @@ Python binding). On Debian/Ubuntu: `apt-get install graphviz`.
 - `simulatorType = Grid2OP` (exact casing) in config.ini; `Pypownet` and `RTE`
   are alternative literals checked in `main.py`.
 - Naming is mixed: public API uses both `snake_case` (`get_dataframe`) and
-  `camelCase` (`isAntenna`, `isDoubleLine`, `getLinesAtSubAndBusbar`). Preserve
-  existing names when editing to avoid breaking the abstract contract.
-- `core/alphadeesp.py` and `core/network.py` rely on `from elements import *`;
-  adding new element classes requires keeping that star import working.
-- Several modules print directly to stdout; there is no logging framework
-  wired up.
-- Only one type-annotated function exists in the codebase — do not expect mypy
-  to be useful without first adding annotations.
+  `camelCase` (`isAntenna`, `isDoubleLine`, `getLinesAtSubAndBusbar`), plus the
+  PascalCase-with-underscores `Structured_Overload_Distribution_Graph`.
+  Preserve existing names when editing to avoid breaking the abstract contract
+  and external importers.
+- `core/elements.py` classes are now imported **explicitly** (e.g.
+  `from alphaDeesp.core.elements import Consumption, Production`); the old
+  `from elements import *` star imports have been removed.
+- Logging: newer modules (`graphs/`, `alphadeesp.py`, `simulation.py`,
+  `elements.py`, `main.py`) use the `logging` framework; the older backends
+  (`grid2op/`, `pypownet/`, `network.py`, `printer.py`) still `print()` in
+  places. Prefer `logging` in new/edited code.
+- Typing: the `graphs/` package, `simulation.py`, `elements.py`, `network.py`
+  and `alphadeesp.py` carry type annotations. CI enforces mypy **strictly** on
+  `simulation.py` and `elements.py` (permissive elsewhere). Do not reintroduce
+  the "codebase is untyped" assumption.
+- `graphs/graph_utils.delete_color_edges` accepts a single colour **or an
+  iterable of colours** and removes them in one graph copy — prefer the
+  multi-colour form to avoid chained full-graph copies.
+- `Structured_Overload_Distribution_Graph.find_loops` can bound simple-path
+  enumeration with `loop_path_cutoff` to avoid hangs on very large grids, but
+  it is **opt-in**: the default is `None` (unbounded = original behaviour)
+  because rustworkx `cutoff` counts *nodes* and real grids have loop paths well
+  beyond any small bound — pass an int only where enumeration is a problem.
+  Consolidation path enumeration has an analogous opt-in
+  `DEFAULT_CONSOLIDATION_PATH_CUTOFF` (also `None`).
+- `OverFlowGraph` copies the caller's DataFrame in `__init__` (it never
+  mutates the frame you pass in).
 
 ## Branch Policy for Claude Code Sessions
 

MANIFEST.in

@@ -7,4 +7,5 @@ include alphaDeesp/ressources/parameters/l2rpn_2019/config.py
 include alphaDeesp/ressources/parameters/l2rpn_2019/difficulty_levels.json
 include alphaDeesp/ressources/parameters/l2rpn_2019/grid.json
 include alphaDeesp/ressources/parameters/l2rpn_2019/grid_layout.json
-include alphaDeesp/ressources/parameters/l2rpn_2019/prods_charac.csv
+include alphaDeesp/ressources/parameters/l2rpn_2019/prods_charac.csv
+include alphaDeesp/core/interactive_html/assets/*