Game Mode: offline pipeline for the France RTE Matpower dataset - #201
Conversation
Adds scripts/game_mode/matpower/, the offline tooling that turns the public MATPOWER RTE cases (case6468/6470/6495/6515rte — real 2013 French EHV operating points) into a Game Mode scenario family alongside the RTE7000 THT one. The cases import as BUS_BREAKER with zero switches, so the expert recommender has no topological levers and grades almost everything hard. node_breaker.py rebuilds the same electrical network as NODE_BREAKER — busbar sections, feeder bays and closed `*_COUPL.*` couplers whose opening is an open_coupling action. Two invariants keep the rebuild faithful: * each substation keeps its loaded electrical node count — the import leaves 208 VLs holding more than one bus (up to 9), and collapsing them onto one busbar rewires the grid (~610 MW extra losses, 26 deg angle shifts, base peak 324% vs 199%), so every source bus gets its own busbar and couplers between distinct nodes are created open; * out-of-service elements stay out — ~700 of 1389 generators carry STATUS = 0 and the bay helpers connect every feeder, and that phantom generation alone stops the load flow converging. Shunts, phase tap changers, generator reactive limits, the slack terminal and the solved (VM, VA) warm start are copied too; each is individually required. On case6515rte the rebuild reproduces the source exactly: 6515/6515 buses, base peak 199.2% / 10 overloads, converged, 1591 coupler breakers. geo.py recovers a France layout for the anonymised cases via the grid_snapshot_reconstruct Rosetta electrical-distance match against a named THT snapshot. That gives a genuine identity mapping for 520 of 6515 buses onto 125 real RTE substations (380 kV only — Rosetta matches the 400 kV backbone), so the rebuild can replicate those substations' real RTE busbar structure (430 VLs). Buses below 380 kV are positioned plausibly but carry no identity claim. grade.py mirrors the THT difficulty rule (unitary / pair / neither) at monitoring_factor 0.95, resetting the recommender before each contingency because run_analysis_step2 mutates network state. Pipeline only: the packaged scenario database, the generated frontend presets and the third Game Mode mode are follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
… third mode Stage 1 (the NODE_BREAKER rebuild) landed in e379ff3 as "pipeline only". This completes the chain: grading, the packaged scenario database, the frontend presets, the transport, and the third Game Mode selectable next to the demo grid and France THT. ## The grading bug that had to be fixed first `game-mode-matpower.md` already stated that the grader MUST reset the recommender before every contingency, because `run_analysis_step2` mutates network state. `grade.py` did not: `configure()` ran once, before the loop. Grading the same 12 contingencies of `grid_6be3a179` (case6515rte) both ways gives **9 divergent verdicts**. The first three agree, then every remaining case collapses to `trivial` with zero overloads. The failure is silent and in the worst direction — a real `hard` scenario vanishes from the database as "nothing to solve" instead of erroring. Grading the 901 contingencies with the committed code would have produced a database missing most of its content. `reset_each` is therefore the default, not an opt-in; `--no-reset` exists for raw timing and its help text says it produces wrong verdicts. It costs ~4x (2.4 s -> 10.6 s per contingency on that grid, ~14 s across the family). That bug also explains the "~86 % medium/hard" prediction recorded earlier in the feature doc: it was measured through the poisoned loop. The full graded distribution is far more balanced — see below. The doc is corrected. ## The full graded database All **901** non-antenna constraining contingencies of the four cases are graded (165 / 270 / 265 / 201 for case6468 / 6515 / 6495 / 6470), ~3.5 h of compute, resumable via the committed per-grid `graded.jsonl`. Of them **870 are playable scenarios**: **428 easy / 379 medium / 63 hard**; 27 `trivial` and 4 `non_converged` are correctly dropped. That balance (~49 / 44 / 7 %) is what makes three real tiers viable, and confirms the corrected doc figure. ## New stages * `build_scenarios.py` (stage 3) folds every per-grid `graded.jsonl` into `data/rte_matpower/scenarios.json`, in the SAME schema as the RTE7000 THT database so downstream consumers read one shape for both families. Only playable verdicts are kept. Scenario ids derive from `(gridId, contingency)`, so a rebuild is stable and does not orphan sessions or retained solutions. * `gen_matpower_presets.py` (stage 4) emits the player-safe `matpowerScenarios.json` + `matpowerPresets.ts` — no reference solution, no date, data kept out of the source module for the line-count gate. * `pack_grids.py` encodes `network.xiidm` -> `.gz.b64` for commit (8.7x: 20.5 MB -> 2.4 MB); `decode_tht_grids.py` now decodes BOTH families (its name is kept because the Dockerfile and docs call it), and the decoded network is gitignored as the build artifact it is. * `grade.py` also records the recommender's own `overloaded_lines` (what the session is judged against) and takes `all` / several grid ids. ## actions.json is the curated source, not the runtime-expanded file `recommender_service.update_config` auto-generates a `disco_` action per line and WRITES THEM BACK into the action file. Pointed at the committed `actions.json`, that bloats it ~8x (curated `open_coupler_*` ~450 KB -> ~3.7 MB of derived `disco_`) and dirties the tree on every grade — and an earlier partial grade had already committed two grids that way. `grade.py` now seeds a gitignored `actions.runtime.json` copy and lets the backend expand THAT, leaving the committed source pristine. All four `actions.json` are re-emitted curated (open_coupler only), removing ~6.6 MB of derived data from the repo. ## Frontend `GameConfigScreen` gains a third mode. Rather than a third copy of the branches, the two graded families are described by one `GRADED` table and the screen branches on "is this a graded family" — level picker, case count, summary, preview and validation are shared. `data-testid`s derive from the family key, so every existing France THT id (and the tests asserting them) is unchanged. The seeded sampler is now shared in `sampleScenarios.ts` and used by both generated preset modules instead of being emitted twice; the RTE7000 module is regenerated onto it (-21/+5, scenario JSON byte-identical, its 12 tests green). Preview map generated from case6515rte's layout. ## Tests 15 Python tests (`test_matpower_game_mode.py`: builder logic on a synthetic family, the player-facing projection, transport decodability, database/frontend consistency), 9 frontend preset tests (no pool-size snapshot — the year-leak check targets the title, since the numeric `LINE-<bus>-<bus>` ids legitimately carry bus numbers in 1900-2099), 3 GameConfigScreen tests for the third mode. 109 frontend tests green, code-quality gate green, ruff clean. End-to-end verified: the committed `.gz.b64` decodes, loads through the real `NetworkService`, and re-running step 1 reproduces a scenario's overloads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…ebuild The node-breaker rebuild only honoured the COUNT of real busbars for mapped substations (clamped by its own logic) and invented the feeder->busbar assignment (each feeder followed its source-bus index; surplus busbars sat empty behind closed couplers). This lands the real thing: scripts/game_mode/matpower/rte_topology.py — detailed-topology plans: - the real voltage level is taken AS IS from the committed named THT reference (grid_5384e039): real busbar count, real cells; - MATPOWER feeders are paired to the real departures (far-site first, exact X/R/B features to split parallel circuits — the Chooz twin lines match to the 2nd decimal and are covered by a unit test); - the target nodal topology = the MATPOWER electrical nodes, and the recommender's nodal->detailed algorithm (expert_op4grid_recommender. manoeuvre, the Python port of RTE's libTOPO) computes which busbar each departure lands on and which couplers stay open to realise those nodes inside the real structure (determiner_topo_complete_cible, replayed with _set_switch and read back via _wired_busbar); - graceful per-VL degradation: incomplete pairing, unverified libTOPO or a missing reference yard simply mean no plan and the generic layout. node_breaker.rebuild_node_breaker(topo_plans=...) joins plans by exact source-bus set; planned VLs get the real busbar count, the libTOPO feeder assignment (unpaired equipment lands on its node group's first busbar) and couplers opened exactly at the node-group boundaries. build_network.py generates plans from the imported network (ids match by construction), preferring the v7 substation map when present. Validated on case6468: 12 VLs planned (9x225, 3x380), rebuild keeps 6468/6468 electrical buses, AC converges, and the 8066-branch flow bench is unchanged vs the analytic bus-branch reference (median |dP| 0.003 MW, p99 0.58, max 11.9 on the same known marginal branch). The MATPOWER node partition is preserved by construction, so fidelity cannot regress. 18 python tests green (3 new), code-quality gate green, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
Generalises the libTOPO real-topology plans from 12 multi-node VLs to every identity-mapped THT voltage level — single-node included — so manoeuvre actions are playable on real substation structures everywhere: - plans are built PER SOURCE VL, keyed by the full pypowsybl bus ids: a multi-node VL's buses share one MATPOWER number, so keying by number merged distinct electrical nodes (measured: 6383/6468 buses, diverging load flow). Bus numbers are recovered from LINE-f-t / TWT-f-t ids; - single anchored node needs NO libTOPO: the real CURRENT wiring of the snapshot is kept (departures on their actual busbars, chain closed) — maximally faithful and still fully splittable; libTOPO runs only when the MATPOWER partition differs (multi-node); - transformers join the pairing (nodes with no line no longer disqualify a VL) — except INTRA-VL transformers, which are series devices (the Logelbach and Creney phase-shifter loops): pairing one to a substation AT cell slots both ends on one busbar and kills it; - the VL's site is the strict buses' COMMON site (loose internal nodes belong to the same physical substation and take part in pairing); - EVERY source node still without a busbar after the libTOPO replay gets its own extra busbar behind an open boundary — unpaired nodes, load/ generator-only buses, and "paired" nodes whose real cell is unwired in the reference state (each of these merged two electrical nodes when missed; all three found by the 6468-bus invariant). Validated on case6468: 1263 planned VLs (972x225, 291x380) = 75 % of THT VLs, 25/93 multi-node (incl. TAVEL on its 3 real busbars, the LOGEL/MUHLB phase-shifter substations), 6468/6468 electrical buses, AC converged, flow bench byte-identical to the generic rebuild (8066 branches, median |dP| 0.003 MW). Playability: 18717 couplers (8376 closed/openable, 10341 open/reclosable) vs 13022 generic (+44 %). Gates: 18+16 python tests green, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…3 multi Iterates on mapping quality and pairing until the multi-node population is (almost) resolved, per the deduction-elimination programme: - TARGET FIX (the big one): unpaired real departures now MERGE into the largest MAT group instead of forming their own node — on every 2-busbar post with 2 MATPOWER nodes the 3-node target was unverifiable, which explained ~38 of the 68 failures. They are not recreated in the rebuild, so the merge is electrically free. - DIRECT-ASSIGN fallback when libTOPO cannot verify (coupling components the port truncates, e.g. CPNIE's 4-SJB coupler): each MAT group laid on its own real busbar, surplus groups fall through to extra busbars. - SITE ARBITRATION by the VL's own ouvrages: conflicting strict sites, or loose-only VLs, are scored per candidate by how many of the VL's lines pair (far site + exact X/R/B) against the candidate yard's real departures. Candidates include — graph structure — every reference site adjacent to the VL's strict far-sites (a VL must be a real neighbour of its neighbours). Winner needs >=2 paired lines or one near-exact match (<0.2), strict dominance required; count-by-type (|k lines - yard size|) is the final tie-break (demoted from 2nd criterion: 2013<->2021 degree drift made it noisy, -57 plans). - RECURSION to fixed point: each planned VL pins its buses' site in an overlay; neighbouring VLs' far-sites gain information and are retried (measured: 1432 -> +34 -> +3 over three passes). case6468: planned VLs 1263 -> 1469 (87 % of THT), multi-node 25 -> 74/93, playable couplers 18717 -> 21318 (+64 % vs generic). Invariants held at every step: 6468/6468 electrical buses, AC converged, flow bench identical (8066 branches, median |dP| 0.003 MW). The 19 remaining multi VLs are one anchor-free cluster (eastern zone / P.AND residue of the former giant codes): no strict identity on any bus and lines pointing at other unresolved VLs — unreachable by local deduction; they need a joint cluster match against the reference (or a better v7 map in that zone) and keep the generic per-node layout meanwhile. Gates: 34 python tests green, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
CLUSTER SOLVER (constrained backtracking on graph structure): local deduction cannot start inside a cluster of VLs that only point at each other. Jointly the constraints are strong — every VL must sit on a site whose yard exists at its voltage, real-adjacent to its resolved neighbours AND to its co-cluster neighbours; (site, kv) yards are exclusive; candidates scored by ouvrage pairing; most-constrained-first DFS, leaving a VL unassigned preferred over failing the cluster. Wired as a final pass after the recursive ones: 35 VLs assigned, +15 plans (TOLBI, FESSH, FLAND... names the map had missed), total 1484-1486 THT VLs (88 %). Invariants held: 6468/6468 buses, AC converged, flow bench identical. MULTI-NODE CLOSURE: the 19 remaining multi-node VLs are NOT physical substations — they are MATPOWER modelling artefacts (3-winding-transformer star buses: TWT cascades 5995-5996-5997; self-referencing TWT on VL-6238; composite VLs aggregating secondary buses of neighbouring posts). Their "real topology" does not exist in the reference; the generic per-node layout is the correct treatment. All identifiable physical multi-node substations are therefore covered (74/93, remainder artefactual). LIBTOPO SURVEY (docs/features/libtopo-port-limits.md): systematic 2-node split bench over the 1383 THT VLs of the named reference. 436/653 posts with >=2 busbars verify (67 %). Two port limitations isolated for the upstream backlog: (1) coupling components >2 SJB truncated by cellules.py (362 VLs emit the warning, 107 failures; ring couplings/transfer bars need the full SJB list and multi-busbar re-routing in the sequencer); (2) non-reroutable departures leaving orphan nodes on 110 un-truncated 2-busbar posts (mono-SA cells/piquages; per-departure diagnostics needed in ResultatManoeuvres.ecarts). 728 mono-busbar "failures" are expected physics, not defects. Gates: 34 python tests green, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: marota <amarot91@gmail.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…détaillée) Utilise l'identificateur libTOPO (LibTopoIdentificateur, phase A) pour générer les plans : seule la correspondance d'ÉTAT nodal -> détaillé compte ici, pas la licéité de la séquence de manœuvres (phase B/C). Replis ImportError/TypeError pour compatibilité avec les libs antérieures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…ie détaillée Le mode jeu ne transforme plus le réseau : toute la chaîne de reconstruction (conversion MATPOWER, limites de courant, calibrage Q, point AC, ré-identification Rosetta, structure de poste RTE réelle, topologie détaillée node/breaker via libTOPO) est désormais dans grid_snapshot_reconstruct, qui produit par case network_detailed.xiidm + grid_layout.json + rte_substation_map.json + detailed_report.json. - build_network.py ne fait plus que ce qui est spécifique au JEU : identité opaque de la grille (la date réelle reste privée), copie des artefacts, espace d'actions offert aux joueurs, screen N-1 base-relatif. Le répertoire source est configurable (MATPOWER_DETAILED_DIR) et un message d'erreur donne la commande exacte qui produit l'instantané manquant. - suppression de node_breaker.py, rte_topology.py, current_limits.py et geo.py (déplacés, cf. grid_snapshot_reconstruct/matpower_*.py) ; - test_matpower_game_mode.py : les tests des parties pures d'appariement partent avec le module (repris dans test_matpower_detailed.py côté reconstruction) ; les 15 tests du mode jeu restent. Vérifié sur l'instantané produit par la nouvelle chaîne (case6468rte) : chargement 6468 nœuds / 86889 organes, artefacts copiés, espace d'actions régénéré à l'identique. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…és en topologie détaillée Les 4 grilles sont réassemblées à partir des instantanés produits par la nouvelle chaîne grid_snapshot_reconstruct (structure de poste RTE réelle jusqu'en 63 kV), puis re-gradées et re-scénarisées de bout en bout. Contrôles de cohérence : - contingences N-1 contraignantes IDENTIQUES à l'unité près avant/après (165, 201, 265, 270 sur les mêmes totaux) — l'augmentation est purement structurelle, le comportement électrique n'a pas bougé ; - espace d'actions par grille : 1557 -> ~8000 couplages manœuvrables (+412 %), effet direct des postes en structure réelle ; - 901 contingences re-gradées (l'ancienne gradation ne valait plus : elle connaissait 5x moins de leviers), 0 sans verdict ; - 870 scénarios jouables : easy 440, medium 368, hard 62 ; écartés 27 triviaux et 4 non convergés ; - bundle joueur : aucune fuite de solution, d'année ni de nom de case ; 9 tests frontend verts. Réserve : 84 traces alphaDeesp (get_dispatch_edges_nodes, TypeError numpy) rattrapées en interne pendant la gradation — la découverte d'actions a pu être partielle sur les contingences concernées, sans jamais empêcher un verdict. Sauvegarde de l'état précédent dans data/rte_matpower/_backup_2026-07-27/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
…t (.gz.b64) Le nouveau build_network.py écrit network.xiidm en clair ; le bundle joueur attend la forme compressée versionnée (cf. rte7000Presets.test.ts). Les quatre réseaux augmentés sont re-packés : 32 Mo -> 3,2 Mo chacun (x10). Signed-off-by: Antoine Marot <amarot91@gmail.com>
… 62→20, +21 easy +21 medium La découverte d'actions plantait sur ``get_dispatch_edges_nodes(only_loop_paths= True)`` quand le DataFrame de boucles rouges est vide (pandas ``.sum()`` renvoie le scalaire 0.0 → ``TypeError: 'numpy.float64' object is not iterable``). Le garde-fou amont ne couvrait que le mode antenne ; corrigé en amont dans Expert_op4grid_recommender (commit a651a40 : test de la condition réelle « aucune boucle rouge exploitable » + try/except). Effet mesuré : sur les 901 contingences, 42 étaient touchées (leur découverte d'actions était tronquée → difficulté surestimée). Re-gradation complète des 4 grilles avec le correctif : 42 transitions, TOUTES hors de « hard » : - hard → easy : 21 - hard → medium : 21 - aucune dans l'autre sens. Répartition jouables (identité des 870 scénarios préservée, mêmes ids ; seuls les paliers changent) : easy 440→461 | medium 368→389 | hard 62→20 (+ trivial 27, non_converged 4) Par grille (hard) : 52bf4231 6→3, 6be3a179 21→6, b1e24f79 10→7, ca0a1d68 25→4. Réseaux inchangés (aucune reconstruction, re-pack .gz.b64 inutile). Bundle joueur régénéré (matpowerScenarios.json 461/389/20, aucune fuite de solution, d'année ni de nom de case). Tests frontend : 109/109 verts. Nota : le correctif vit dans Expert_op4grid_recommender (a651a40, branche claude/libtopo-95pct). Le venv de ce repo porte le même patch chirurgical (à répercuter proprement au prochain bump de la dépendance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Antoine Marot <amarot91@gmail.com>
e379ff3 to
699f971
Compare
Mise à jour — dataset re-gradé après hotfix alphaDeesp + bump de dépendanceBranche mise à jour (head Bump de dépendance
Dataset de difficulté re-gradéLe bug plantait
Fichiers
Tous les commits sont signed-off (DCO). |
Consumes the alphaDeesp empty-red_loops hotfix (ainetus/Expert_op4grid_recommender#124, tag v0.3.3.post1). The Matpower difficulty dataset in this branch was re-graded with that fix — 42 scenarios reclassified out of hard (hard->easy 21, hard->medium 21), tier split 440/368/62 -> 461/389/20. CHANGELOG updated. Signed-off-by: Antoine Marot <amarot91@gmail.com>
699f971 to
2171339
Compare
Adds
scripts/game_mode/matpower/— the offline pipeline that turns the public MATPOWER RTE cases (case6468/6470/6495/6515rte, real 2013 French EHV operating points) into a Game Mode scenario family alongside the RTE7000 THT one.Pipeline only. The packaged scenario database, the generated frontend presets and the third Game Mode mode are follow-ups.
Why a node-breaker rebuild
MATPOWER cases import as
BUS_BREAKERwith zero switches, so the expert recommender has no topological levers (no coupler opening, no node splitting) — only redispatch and load shedding. On these heavily loaded states that grades almost everything hard.node_breaker.pyrebuilds the same electrical network asNODE_BREAKER: busbar sections, feeder bays, and closed*_COUPL.*couplers whose opening the recommender picks up as anopen_couplingaction.Two invariants keep it faithful — both found by diffing against the source, not by guesswork:
STATUS = 0, and pypowsybl's bay helpers create every feeder connected — that phantom generation alone stops the load flow converging.Shunts, phase tap changers, generator reactive limits, the slack terminal and the solved
(VM, VA)warm start are copied too; each is individually required.On
case6515rtethe rebuild reproduces the source exactly: 6515/6515 buses, base peak 199.2% / 10 overloads, converged, 1591 coupler breakers (266 open / 1325 closed).France positioning + real RTE structure
The cases are anonymised (integer buses, no names, no coordinates).
geo.pymatches each case's 400 kV postes to a named THT snapshot via thegrid_snapshot_reconstructRosetta electrical-distance percolation, then chains togrid_layout_rte.json.That yields a genuine identity mapping for 520 of 6515 buses → 125 real RTE substations (380 kV only — Rosetta matches the 400 kV backbone). Where a bus is identified, the rebuild replicates that substation's real RTE busbar count (430 VLs on
case6515rte, giving real 4-, 6- and 9-busbar substations). Buses below 380 kV are positioned plausibly but carry no identity claim — documented as such.Grading
grade.pymirrors the THT rule atmonitoring_factor = 0.95(unitary → easy, pair → medium, neither → hard), base-relative. It resets the recommender before every contingency, becauserun_analysis_step2mutates network state and grading in a loop otherwise silently poisons every subsequent contingency.case6515rteyields 270 non-antenna constraining contingencies of 7422 tested, at ~11.5 s each.Notes
grid_<sha1[:8]>folders, month + weekday + hour-period titles);mapping_private.json/rte_substation_map.jsonkeep identity recoverable for analysis and are never surfaced to players.case6468rte85 GW vscase6515rte107 GW).ruffandscripts/check_code_quality.pyboth pass. No dataset bytes in this PR (/data/is gitignored).🤖 Generated with Claude Code