Skip to content

Improve performance - #336

Open
bl-young wants to merge 2 commits into
developmentfrom
by_memory
Open

Improve performance#336
bl-young wants to merge 2 commits into
developmentfrom
by_memory

Conversation

@bl-young

@bl-young bl-young commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

This resolves some challenges I was having generating the 2023 baseline from the current develop branch. It includes two main changes that have negligible effect on the final datasets:

  1. In combinator.py, adds _build_flowmapping_small_for_merge() for generating a more manageable flow mapping file which preserves intentional eLCI 1:many splits (e.g. sand/gravel) but doesn't blow up the mapping when pulling in the rest of the fedefl (via Unexpected mis-matched flows in eLCI.csv #274). This helped me avoid Memory Errors

  2. In generation.py, replace groupby(...).apply(_combine_sources) with vectorized helpers to improve performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant