Separate validation from parsing in model and study modules - #290
Closed
aoustry wants to merge 5 commits into
Closed
Separate validation from parsing in model and study modules#290aoustry wants to merge 5 commits into
aoustry wants to merge 5 commits into
Conversation
* Add first tests on thermal heuristics * Implement accurate heuristic on simple problem * Add comments * Add fast heuristic and improve accurate * rename test heuristic * Implement complex model * Accurate for complex model * Type errors and useless solver parameters * check hourly outputs * fast heuristic for complex case * Test for 2 weeks * Test for 2 scenarios * add test for scenarios * Add more details on tests concerning scenarios * Add milp test for second case * remove milp test * Add test accurate for second test case * Add fast test for second test case * convert fast to optimization problem * Optimization problem for fast on simple case * Optimization model fast complex case * Small corrections * Correct test * Optimization problem heuristic fast second complex test case * Equivalent solution for fast heuristic optimization * Api for fast and accurate models * Api for accurate heuristic model * Move models to library * Move some functions from tests to src * Remove all functions from simple test * Move functions from second complex test to src * Remove functions from second test and filter ts on scenarios and timesteps * Sort imports * Run black * Change solver to fix tests in the ci * Fix ci * Rename tests * Generalize check_output * More tests on data * Refactor model * Refactor problem.py * Generic database * Correct error * Create ThermalProblemBuilder * Resolution steps * Simplify get_value out of database * Simplify update of database * Fix ci * Remove not used imports * New test * Small changes * New test with bc * Refactoring * Exhibit models * Remove unused imports * New class for time scenario parameters * Refactor solve * Sort imports * Formating * Refactor data_path * Fix test * Refactor models * Fix test * Run isort * Week scenario parameters * Refactor solve * Rename functions * Refactor cluster parameters * Correct error * Refactor update * New test * New test with ramp * Refactor * Add description for tests * Beginning of new test * Improve edit_value * Improve cluster parameters * Move expected_output class * Change names * Remove network and database building from thermal_problem_builder * Correct pytest fixture * Remove solve from thermal problem builder * Fix test * Move tests * Improve test with parameters * Use data_path * Improve pytest fixture * Improve tests * Fix ci * Fix tests * Fix test on day ahead reserve * Unused code * api file * Small changes * Move files * Prepare test_one_cluster * Update tests * Update test with bc * Update test with different scenarios and xpress settings * Remove test with different scenarios and xpress settings * Remove day ahead test * Implement fast heuristic as an algorithm * Implement accurate heuristic * integer-strategy in system and heuristic in optim-config * Implement interger-strategy in variables building * heuristic id in parsing * Add heuristic validation and update optim-config documentation * Workflow with 2 iterations and thermal heuristic * Remove obsolete code * Fix ci * Test with ramp * Validate heuristic input/output time-dependence at optim-config parsing Cross-check each heuristic input/output declared in optim-config.yml against the model: the referenced id must exist and have the time-dependence the fast/accurate thermal heuristics expect (e.g. min_up_duration constant, generation_power per-timestep), catching mismatches at load time instead of a runtime crash mid-solve. nb_units_max and cluster_max_generation now accept either form, with the heuristics broadcasting a scalar internally. Also warn (instead of silently truncating) when min_up_duration/ min_down_duration resolve to a non-integer number of timesteps. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Validate optim-config automatically in SimulationSession * Fix ci * Refactor e2e tests * Formatting * Refactoring * Fix solution retrieval, window size in fast heuristic bugs * Review comments * non_prop_cost in tests * Remove tests of heuristic behaviour * New mixed strategies test * Enforce heuristic-id consistency with integer-strategy * Guard bound mutation on merged relaxed/exact variables The merged Variable rebuilt for split integer/binary variables was a detached xr.concat copy: setting .lower/.upper on it silently wrote to an orphaned copy instead of the solver, and its name/label_range were inherited from only one of the two groups. Now fails loudly and carries correct metadata. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document integer-strategy and thermal heuristics Documents the new per-component integer relaxation strategy (exact/relaxed/heuristic) and the built-in fast/accurate thermal heuristics in optim-config.md, building.md, AGENTS.md, and the changelog. Also fixes a mypy error-code annotation in optimization.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update changelog * Add lower_bound()/upper_bound() expression operators Surfaces a variable's current bound in extra-outputs and port-field definitions, mirroring dual()/reduced_cost(). This lets heuristic-mutated bounds (e.g. the fast thermal heuristic tightening generation_power's lower bound) be read back post-solve instead of only the solved value. Reading bypasses the detached merged relaxed/exact variable copy the same way get_variable_solution() already does, so mutations reach the output. Use the new operator to fix num_units_on/non_prop_cost in the thermal heuristic model libraries: they previously approximated unit commitment from generation_power / max_power_per_unit, which diverges from what the fast heuristic actually enforces; deriving it from lower_bound(generation_power) / min_power_per_unit matches it exactly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix tests --------- Co-authored-by: Juliette-Gerbaux <juliette.gerbaux_externe@rte-france.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Thomas Bittar <thomas.bittar@rte-france.com>
* New test with no pmin * Fix fast heuristic to track upper bound when min_power_per_unit is 0 Previously find_min_generation_fast short-circuited to all zeros whenever min_power_per_unit was ~0, dropping the unit-commitment info needed to keep committed units available for production. It now always computes num_units_on and returns both a minimum_generation_power and a maximum_generation_power series, with the upper bound wired through the heuristic outputs schema and the one-cluster no-pmin study.
…nto claude/pr-278-tbittar-comments-6yn041
Addresses tbittar's review on #278: cross-artifact validation had been put inside parse_yaml_library, which is the reading/validating mixing that #265 set out to undo. - New gems_craft/model/validation.py holds check_library_against_taxonomy (moved unchanged from taxonomy.py, with its _missing helper) plus validate_libraries_against_taxonomy, which absorbs the declared-id and missing-taxonomy checks. - parse_yaml_library goes back to a pure reader: no taxonomy argument, no validation. input_libs likewise loses the argument. - load_study reads input/taxonomy.yml and calls the validation explicitly after parsing, alongside consistency_check. - taxonomy.py no longer imports parsing, so the TYPE_CHECKING guard added to break that cycle is gone — the cycle no longer exists. - Acts on the TODO left by #265: consistency_check moves from resolve_components.py to a new gems_craft/study/validation.py; its callers are repointed and a now-dead Model import is dropped. Conformance is enforced when a study is loaded rather than on every parse, matching how consistency_check and validate_optim_config already work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGFvLY7afUMBnwfHgKHHAD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Process ID
Process: N/A
Description
This PR refactors validation logic to be separate from parsing, mirroring the structure of
optim_config/. The changes improve code organization by creating dedicated validation modules while keeping parsing pure.Key Changes
Model Module:
gems_craft/model/validation.pywith taxonomy validation logic:validate_libraries_against_taxonomy(): Validates libraries declaring a taxonomy against the provided taxonomycheck_library_against_taxonomy(): Validates individual library models against taxonomy categories_missing(): Helper to identify missing required fieldscheck_library_against_taxonomy()fromtaxonomy.pytovalidation.pyTaxonomyDatatoTaxonomySchemaintaxonomy.py(no alias kept)taxonomy.py, keeping it focused on data structures and loadingStudy Module:
gems_craft/study/validation.pywithconsistency_check()functionconsistency_check()fromresolve_components.pytovalidation.pyload_study()infolder.pyto:input/taxonomy.ymlif presentvalidate_libraries_against_taxonomy()on all libraries declaring a taxonomyValueErrorif a library declares a taxonomy but none is provided, or if taxonomy IDs don't matchBreaking Changes:
check_library_against_taxonomyimport path changed fromgems_craft.model.taxonomytogems_craft.model.validationconsistency_checkimport path changed fromgems_craft.study.resolve_componentstogems_craft.study.validationTaxonomyDatarenamed toTaxonomySchemawith no backward compatibility aliasBehavior Changes
load_study()now validates libraries against the study's taxonomy (if provided)parse_yaml_library()remains unchanged—no validation occurs during parsingImpact Analysis
Affected modules:
gems_craft/model/— Validation separated from taxonomy modulegems_craft/study/— Validation separated from resolve_components moduleSolver output: No change expected. This is a refactoring of validation logic with no impact on simulation results.
Checklist
validate_libraries_against_taxonomy)CHANGELOG.mdupdated with breaking changes and new featuresAGENTS.mdreviewed and updated