[FEATURE] Add utility function to reorganize scenarios according to the performance guide

[mgovers]

Describe the problem

Different use cases require different sets of calculations. Since the PGM treats scenarios in an isolated fashion, the core is agnostic to the order of calculations.

However, some caching may happen in the PGM core to speed up the calculations. But whether and what can be cached depends on the order and contents of the scenarios in the update data, as well as amount of threads used during the calculation.

Guidelines to speed up the calculation are documented in https://power-grid-model.readthedocs.io/en/stable/user_manual/performance-guide.html#batch-calculations .

For many use cases, the order of scenarios does not really matter to the user (e.g., when doing N-1 simulations). Even when the order does matter (e.g., when doing time series calculations), only the final order matters.

Improvement proposal

• Add a function reorder_scenarios_for_performance(*, BatchDataset update_data) -> tuple[BatchDataset, list[int]] to the power_grid_model.utils module that:
  • creates an ordering reordering: list[int] of the scenarios (following the conventions of np.argsort) in the update data according to the performance guidelines, that is:
    • it keeps scenarios with the same topology (e.g., same switch statuses) together
    • within the same topology, it keeps scenarios with the same electrical parameters (e.g., branch parameters, tap position, ...) together
  • creates a copy (!!!) of the update data with the reordered order of scenarios, such that it is equivalent to

    for reordered_idx, scenario_idx in enumerate(reordering):
        np.testing.assert_array_equal(
            get_dataset_scenario(reordered_update_data, reordered_idx),
            get_dataset_scenario(update_data, scenario_idx)
        )
    # for row-based dense batch data, this is equivalent to:
    for component in update_data:
        np.testing.assert_array_equal(
            reordered_update_data[component][:],
            update_data[component][reordering]
        )

  • computes the inverse_ordering: list[int] of the scenarios, such that

    for scenario_idx, reordered_idx in enumerate(inverse_ordering):
        np.testing.assert_array_equal(
            get_dataset_scenario(reordered_update_data, reordered_idx),
            get_dataset_scenario(update_data, scenario_idx),
        )
    # for row-based dense batch data, this is equivalent to:
    for component in update_data:
        np.testing.assert_array_equal(
            reordered_update_data[component][:],
            update_data[component][inverse_ordering],
        )

  • returns both the reordered_update_data and the inverse_ordering
  • works for both row-based, columnar data, dense and sparse data, as well as a mixture of the two
  • has a clear docstring
• Add tests for row-based, columnar, dense, sparse and mixed data.

[appleweiping]

Hi, I would like to work on this next, now that #1540 has been merged, if it is still available. I reviewed the current main, the batch data layouts, the performance guide, and the related #1103. Before implementing, could you please confirm a few points where the intended contract is not yet clear to me?

1. Relationship with [IMPROVEMENT] Combine reverse update data and new update data to more cleverly cache #1103. The current batch path restores the model to its base state after every scenario, and restore_components() invalidates topology/parameter caches after a non-base update. It therefore appears that grouping identical non-base topology/parameter states only provides the intended cross-scenario cache reuse once [IMPROVEMENT] Combine reverse update data and new update data to more cleverly cache #1103 combines the previous inverse update with the next scenario update. Should [IMPROVEMENT] Combine reverse update data and new update data to more cleverly cache #1103 be implemented first, or is [FEATURE] Add utility function to reorganize scenarios according to the performance guide #1100 expected to be useful and merge independently?

2. Equivalence key. This API receives only update_data, not input_data or a model. It cannot know that an omitted/NA value and an explicitly supplied value happen to produce the same effective state relative to the input. Should the utility conservatively group scenarios by their literal update-patch signature?

3. Topology/parameter classification. The Python metadata describes data shapes and attributes, but does not expose the C++ UpdateChange topology/parameter categories. Would you prefer focused Python classification tables that mirror the component update() semantics, or should this metadata first be exposed/generated from a single source so the two layers cannot drift?

4. Ordering direction. May I interpret the contract as:

   • reordered = original[reordering];
   • inverse_ordering = np.argsort(reordering);
   • original == reordered[inverse_ordering];
   • scenarios with equal topology and parameter keys retain their original relative order?

   The per-scenario inverse example appears to use this convention, while the dense inverse example currently appears reversed.

5. Representation/copying. I plan to preserve each component's existing row/columnar and dense/sparse representation, allocate independent copies of all arrays, and for sparse data reorder whole scenario blocks while rebuilding indptr. Is that the desired copy contract?

If these assumptions are confirmed, I would first add focused tests for row-based dense, columnar dense, row-based sparse, columnar sparse, mixed layouts, stable ties, empty scenarios, input immutability, and inverse restoration, and then implement the smallest public utility in power_grid_model.utils.

I will use OpenAI Codex interactively for repository navigation, analysis, test design, implementation, and verification. I will personally review and understand every change, remain responsible for the contribution, use signed/DCO-compliant commits, and disclose the assistance in the PR.

[nitbharambe]

Hi @appleweiping , nice that you are picking this up.

1. Good call. I imagine the resultant ordering output would be different in both situations. However there is no target date for that feature yet. I think it would be best if we keep both issues (This and [IMPROVEMENT] Combine reverse update data and new update data to more cleverly cache #1103) separate. (Unless you or someone else wish to pick that issue up as well, then situation changes! :D). We should also highlight in the docs / docstring that output of this function may change as per the latest improvement / features.

2. No issue in adding input_data keyword arg in my opinion. And we can set it optional and conservatively group assuming the input_value != nan and input_value != update_value.

3. Maybe we can start this implementation with hardcoded params. Lets scope this out. An ideal solution can be related to [IMPROVEMENT] PGM Attributes management over all projects #1519.

4. Yes what you mention should be the behaviour. The dense inverse example is incorrect. Ill modify it such that inverse_ordering applies on the reordered_update_data such that we can construct original input data.

5. Sounds good.

[TonyXiang8787]

@mgovers @nitbharambe so your proposal is only a Python convenienc function instead of C-API core feature?

[nitbharambe]

@TonyXiang8787 Indeed. We did not receive any specific request from C API users as of yet. Hence we decided to start with a python utility initially as it is easier to implement.

[nitbharambe]

@appleweiping Since its been an year this issue was posted, please hold on the implementation for some time while we discuss requirements further within the users / maintainers.

[appleweiping]

Thanks for the clarification. I’ll hold off on implementation while the users and maintainers refine the requirements. I’ve noted the current direction—an initial Python utility, separate from #1103, with optional input data, focused hardcoded classification, and the confirmed ordering/inverse-ordering contract—but I won’t treat it as final or start coding until you confirm the scope is ready. Please tag me when the discussion has converged and I’ll be happy to continue.