Streamline workflow

[FabianHofmann]

Closes #1916. (supersedes)

Streamlined and restructured workflow

This PR refactors the workflow in order to streamline and unify optimization approaches and configuration settings. here are some highlights of the PR, otherwise no one will have the motivation to look at this large change. I suggest before going into the code, read this here first, even though it is long...

Highlights

• No more wildcards excepts for {horizon}; all moved explicit to config entries
• No more cryptic filenames like elec_s_37_lv1.25_3H_2030.nc
• Processing of network follows the logic of base.nc -> clustered.nc -> composed_{horizon}.nc -> solved_{horizon}.nc
• Results are unified and made independent of horizons (csvs/nodal_costs.csv, csvs/capacities.csv ...)
• compose rule is the entry point for green-field, brown-field, perfect foresight model for both sector-coupled and electricity-only.
• Electricity-only models now run with myopic an perfect foresight
• Make and plot summary now works for all combinations of models (elec/sector) and foresights

and now a more descriptive summary

Overview

• The pr restructures the Snakemake workflow of PyPSA-Eur into a leaner pipeline (base → simplified → clustered → composed → solved) and standardises filenames so scenario information now lives in configuration instead of wildcards.
• All rules that previously emitted *_s_{clusters} or ..._{planning_horizons} artefacts have been renamed; the new naming scheme makes horizons explicit via {horizon} and removes cluster suffixes.
• Post-processing rules were refactored to read the new solved network filenames and now operate uniformly across all foresight modes, so the map and summary targets remain identical for overnight, myopic, and perfect runs for both elec-only and sector-coupled.

Workflow Changes

1. Stage progression – The workflow now moves strictly through networks/base.nc → networks/simplified.nc → networks/clustered.nc → networks/composed_{horizon}.nc → RESULTS/networks/solved_{horizon}.nc. Old intermediate targets such as networks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.nc no longer exist.
2. Unified composition – rules/compose.smk encapsulates what used to be multiple prepare_*, add_*, and brownfield rules; it assembles everything needed for a given horizon and handles the myopic/perfect brownfield inputs.
3. Single solve rule – rules/solve.smk contains one solve_network rule; electricity-only and sector-coupled cases are distinguished by config, not by separate rule files (solve_* smks were deleted).
4. Collection targets – rules/collect.smk now checks four milestones: clustered networks, composed networks, solved networks, and plotting; {horizon} replaces {planning_horizons} wildcards in file names.
5. Scenario-aware config – navigate_config, get_full_config, and get_config (all in rules/common.smk) cache fully merged configs per wildcard set. Any new rules should obtain parameters through config_provider(...) or get_config(w) to stay scenario compatible.
6. Warm-start logic – Myopic runs read the previous RESULTS/networks/solved_{prev}.nc, while perfect foresight reuses the prior networks/composed_{prev}.nc. Overnight runs must supply a single planning horizon.
7. Post-process harmonization – rules/postprocess.smk now reads RESULTS/networks/solved_{horizon}.nc for all foresight modes, so every map (power_network_{horizon}.pdf, h2_network_{horizon}.pdf, balance_map_{carrier}_{horizon}.pdf, etc.) and CSV summary is generated with the same naming scheme regardless of foresight setting.

Stage Notes

• Base & shapes – rules/build_electricity.smk writes onshore_regions_base.geojson/offshore_regions_base.geojson together with networks/base.nc. Administrative shapes and OSM/TYNDP inputs remain unchanged.
• Simplified assets – simplify_network now emits networks/simplified.nc, onshore_regions_simplified.geojson, offshore_regions_simplified.geojson, and busmap_simplify_network.csv. build_electricity_demand_base consumes those files and produces electricity_demand_simplified.nc. process_cost_data reads per-horizon costs_{horizon}.csv.
• Clustering – cluster_network takes networks/simplified.nc plus busmap_simplify_network.csv and emits networks/clustered.nc, onshore_regions.geojson, offshore_regions.geojson, busmap_cluster_network.csv (chained to the final busmap.csv via chain_busmaps), and linemap_cluster_network.csv. Cluster counts are configured, not embedded in filenames.
• Compose assets – Most intermediate files on inputs drop the _s_{clusters} suffix; ie. population layouts collapse to pop_layout.csv and pop_layout_simplified.csv; Gas locations become gas_input_locations.geojson/gas_input_locations_simplified.csv; Powerplant list becomes powerplants.csv etc.
• Composition – compose_network (one rule) flattens all previous add_existing_baseyear, add_brownfield, prepare_perfect_*, etc. It automatically wires the previous-horizon inputs depending on foresight and provides the final pre-solve network (networks/composed_{horizon}.nc).
• Solving – solve_network reads networks/composed_{horizon}.nc, writes RESULTS/networks/solved_{horizon}.nc, and stores solver/memory/python logs under RESULTS/logs/solve_network/. Custom extra functionality and solver settings are driven purely by config.
• Post-processing – Preview maps are maps/base_network.pdf and maps/clustered_network.pdf; solved outputs live under RESULTS/maps/static/ (power_network_{horizon}.pdf, h2_network_{horizon}.pdf, ch4_network_{horizon}.pdf, balance_map_{carrier}_{horizon}.pdf) with interactive HTML equivalents under RESULTS/maps/interactive/. make_summary and plot_summary operate on RESULTS/networks/solved_{horizon}.nc regardless of foresight.

Config Changes

• Planning horizons are now top-level – Set planning_horizons directly under the root of your config (config/config.default.yaml:34-35). The old scenario.planning_horizons entry is ignored, and the workflow expects config["planning_horizons"] to exist even when scenarios are disabled. Configuration bundles such as config/test/config.scenarios.yaml:17-20 already follow this format.
• Global scenario block removed – The legacy scenario: section (with clusters, opts, sector_opts, etc.) is no longer part of config.default.yaml. Scenario sweeps should now be described via run.scenarios plus the dedicated scenario YAML file; individual dimensions (e.g. clusters) are configured directly under their respective sections. If you keep a scenario block in a local config it will simply be ignored.
• CO₂ budget fields were restructured – co2_budget now specifies an emissions_scope, a relative flag (values interpreted relative to a 1990 baseline when true, absolute when false), and nested upper/lower dictionaries of year-value pairs (config/config.default.yaml). Update custom configs accordingly if you previously listed plain year-value pairs.
• Transmission capacity caps gained explicit “extension” keys – lines uses s_nom_max/s_nom_max_extension, and links uses p_nom_max/p_nom_max_extension (config/config.default.yaml:320-355). Rename any overrides that still refer to max_extension so the new limits are applied.
• Selective carrier exclusion moved into configuration – electricity.exclude_carriers (config/config.default.yaml:115-162) lets you strip carriers during clustering; custom busmap logic should now read that list instead of hard-coding exclusions.
• Existing capacities follow the foresight mode – compose_network imports historical assets at the first planning horizon for myopic and perfect foresight; overnight runs assume a greenfield build. The base year for existing capacities is the first entry of planning_horizons (as on master), so there is no separate toggle or base-year key under existing_capacities.
• National phase-outs moved to config – The formerly hardcoded policy phase-outs (German coal 2030, nuclear exits, etc.) now live in existing_capacities.phase_outs as a list of {carriers, countries, year} rules. They cap conventional asset lifetimes and apply to both generators (electricity-only) and links (sector-coupled) in perfect foresight. Defaults reproduce the previous behaviour; edit or empty the list to customise.

Script Changes

• scripts/compose_network.py combines function calls from multiple preparatory scripts (add_electricity.py, add_existing_baseyear.py, add_brownfield.py, prepare_network.py, prepare_sector_network.py, prepare_perfect_foresight.py). At a later stage a clearer packaging structure should be used here.
• Preparatory scripts are no longer executed – there main sections was integrated into compose_network.py.
• Solving entry points are harmonized – scripts/solve_network.py is the only solver script called from Snakemake; solve_operations_network.py (operations-only runs) is no longer referenced by the workflow.
• New scripts/co2_budget.py provides the helpers for the restructured co2_budget configuration (per-horizon bounds and budgets).
• build_clustered_solar_rooftop_potentials.py was renamed to build_solar_rooftop_potentials.py.
• Summary generation moved into scripts/make_summary.py – the script now loads all horizons via pypsa.NetworkCollection for overnight, myopic, and perfect runs, so the dedicated helpers scripts/make_summary_perfect.py and scripts/make_global_summary.py were deleted. Any custom tooling should invoke make_summary.py and read the unified CSV outputs.

File Name Mapping

Old target	New target	Notes
networks/base_s.nc	networks/simplified.nc	Paired with onshore_regions_simplified.geojson, offshore_regions_simplified.geojson, and busmap_simplify_network.csv.
regions_onshore_base_s_{clusters}.geojson / regions_offshore_base_s_{clusters}.geojson	onshore_regions.geojson / offshore_regions.geojson	Produced by cluster_network; no {clusters} wildcard in filename.
busmap_base_s_{clusters}.csv / linemap_base_s_{clusters}.csv	busmap.csv / linemap_cluster_network.csv	Final busmap.csv chained via chain_busmaps; clusters now implicit in config.
powerplants_s_{clusters}.csv	powerplants.csv	Generated after clustering; uses networks/clustered.nc.
electricity_demand_base_s.nc	electricity_demand_simplified.nc	Created from simplified assets.
availability_matrix_{clusters}_{technology}.nc	availability_matrix_{technology}.nc	Includes MD/UA variants.
profile_{clusters}_{technology}.nc	profile_{technology}.nc	regions_by_class_* follow the same pattern.
pop_layout_base_s_{clusters}.csv / pop_layout_base_s.csv	pop_layout.csv / pop_layout_simplified.csv	Solar rooftop potentials drop cluster suffixes too.
gas_input_locations_s_{clusters}.geojson	gas_input_locations.geojson	Simplified CSV also loses the suffix.
costs_{planning_horizons}.csv / _processed.csv	costs_{horizon}.csv / _processed.csv	collect.smk expands over {horizon}.
networks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.nc (+ _brownfield*)	networks/composed_{horizon}.nc	Handles all foresight modes.
RESULTS/networks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.nc	RESULTS/networks/solved_{horizon}.nc	Solver logs renamed accordingly.
maps/power-network.pdf / maps/power-network-s-{clusters}.pdf	maps/base_network.pdf / maps/clustered_network.pdf	Cluster preview plotting renamed.
RESULTS/maps/base_s_{clusters}_{opts}_{sector_opts}-costs-all_{planning_horizons}.pdf	RESULTS/maps/static/power_network_{horizon}.pdf	Same pattern for hydrogen, methane, and balance maps.

TODO

• double check migration guide
• set_all_phase_outs(): National energy policy phase-out constraints (moved to config existing_capacities.phase_outs; generalized to generators + links, applied in perfect foresight)
• update_heat_pump_efficiency(): Heat pump COP updates across all periods (myopic via add_brownfield; perfect via generalized twin-repoint in concatenate_network_with_previous)
• apply_time_segmentation_perfect(): Time segmentation for multi-period networks
• remove unneeded set_investment_periods from code (related to a previous bug in pypsa)
• cluster loads before compose (new cluster_electricity_demand rule → electricity_demand.nc; attach_load no longer maps simplified→clustered)
• double check the usage of busmaps, return final busmap from base to clustered
• should we remove operational solving? No, rewired into rules/solve.smk as opt-in solve_operations_network (solved_{horizon}.nc → operations_{horizon}.nc); rolling-horizon dispatch decoupled via new solving.operations block, enabling capacity expansion followed by rolling dispatch.
• change rule overview in foresight.rst to modules overview (look for similar places)

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in config/config.default.yaml.
• A release note doc/release_notes.rst is added.

[coroa]

Thanks for starting this. Had only a short look so far. I guess it is too early for reviewing. How do you think we can best support?! I have opinions about many of the "critical open questions".

[FabianHofmann]

You are totally right, it is too early for reviewing. I guess at this stage, it is best to think about whether I can proceed with this approach or if there is potential red flags, which speak against this strategy in the first place. But perhaps I spend more time on refining this and make a proper report for you guys what are this wider implications.

[FabianHofmann]

I have opinions about many of the "critical open questions".

I must admit that the section Critical Unresolved Questions in the streamline document might be outdated and partially resolved (and perhaps even incomplete), so don't give it too much attention. but I would still be happy to hear any high level thoughts and critical things that I potentially miss!

[FabianHofmann]

@fneum I still need to do some small things (delete dead code, sanitizing little stuff).

The workflow now reproduces the exact same results as master for all test cases (objective assertion is in place). Plots produce nicely and things run very smooth. so I have the feeling this is good to go soon. I'll ping you next week likely when I am fully happy

[fneum]

@FabianHofmann I'd have some time to review this in the next few weeks. Should we give this a final push over the finish line?

[FabianHofmann]

@FabianHofmann I'd have some time to review this in the next few weeks. Should we give this a final push over the finish line?

You are a hero. Let me merge the latest master then this week and go through it one more time. But the last time I looked I was quite happy.

[fneum]

Nice! We can also do a co-working session if you want.

[FabianHofmann]

Nice! We can also do a co-working session if you want.

I'll see if I can do some preparation tomorrow. but lets meet anyway next week with or without the merge on the todo list.

[coroa]

We are wondering from the open-tyndp side, whether it makes sense to do a trial run soft-fork merge with us to see what we need to do and to test the instructions. Let's have a small chat about this @FabianHofmann @lisazeyen

[fneum]

We should definitely do a release before merging this.