feat(parquet): variables metadata and download API for parquet outputs by sylvlecl · Pull Request #3367 · AntaresSimulatorTeam/AntaREST

sylvlecl · 2026-08-20T15:59:34Z

Done
Implementation is considered complete for area data only (values):

DB models for variables and areas
function for extracting them from file outputs and dump them to database
methods for reading them back from database, using a consolidated
interface IParquetOutputMetadata
implementation of iteration over area dataframes, respecting the variables that are
actually defined for each area
implementation of the "download" API on top of that iteration function

To be completed with

implementation for other element types: links, thermal clusters ...
Pay attention: thermal clusters and similar elements will need some special work to re-shape the
dataframe when writing to parquet (like today in aggregation), and the opposite transformation for
the download API
technical "detail":
merge the "parquet_output" table I added into the existing output metadata table.
The creation needs to be slightly changed to get the integer primary key from the DB
alembic migration

- DB models for variables and areas - functions for extracting them from file outputs to be continued with: - some tests - implement the download API for just areas for now - implement other element types Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

+from functools import cached_property
 from pathlib import Path
-from typing import Callable, Generic, Literal, Sequence, TypeAlias, TypeVar
+from typing import Callable, Generic, Iterable, Literal, Sequence, TypeAlias, TypeVar


Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl · 2026-08-21T15:49:39Z

    raise OutputSubFolderNotFound(output_dir.name, "economy|adequacy")
+
+
+class FileOutput:


helper class to make output directories analyze easier, could be re-used in other places to simplify the code (in particular iteration module of filestudy outputs)

sylvlecl · 2026-08-21T15:52:14Z

+
+    __tablename__ = "parquet_output"
+
+    id: Mapped[int] = mapped_column(BigInteger, primary_key=True, autoincrement=True)


as for study_data_id, the idea here is to introduce a technical ID for outputs, separate from study_id / output_id

This will be beneficial in the future for:

more compact foreign keys in other tables

possibility to rename the output if the user wants it

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl · 2026-08-21T15:59:52Z

+    mc_ind_vars: Mapped[list[int]] = mapped_column(Columns)
+
+
+# TODO: add tables for other element types: links, thermal clusters, etc


Easier to focus only on areas for now, but others will need to follow of course

sylvlecl · 2026-08-21T16:00:38Z


            simulation_range = _extract_simulation_range(dir_path)

+            # TODO: first, extract variables metadata to database


here we'll need to call the new functions from variables_parsing, and provide the variables index to the parquet creation function so that it uses the column that have been defined there

sylvlecl · 2026-08-21T16:02:35Z

+
+
+@dataclass(frozen=True)
+class OutputParsingResult:


We may not need that class if we dump to database the "partial" result one after the other

sylvlecl · 2026-08-21T16:04:35Z

+                yield parse_output_file(data_file, start_col)
+
+
+def _extract_areas(


This is completely to be done:

now we will need to take care of writing the columns in the order that has been defined at variables parsing time, so that the parquet file is consistent with the variable definitions we have stored in database.

It may involve to re-write most of the code in this file unfortunately.

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

        self, study_id: str, output_id: str, data_selection: StudyDownloadDTO
    ) -> MatrixAggregationResultDTO:
-        raise NotImplementedError()
+        metadata = self._require_metadata(study_id, output_id)


+from antarest.study.model import MatrixFrequency
+
+
+def _parquet_file_name(element_type: ElementType, frequency: MatrixFrequency) -> str:


Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl · 2026-08-25T16:55:01Z

+    def get_mc_all_file(
+        self,
+        file_type: MCAllAreasQueryFile | MCAllLinksQueryFile,
+        area_id: str,


should be renamed element_id if re-used for links

sylvlecl · 2026-08-25T16:55:54Z

+ScenarioAggregation: TypeAlias = Literal["mc-ind", "mc-all"]
+
+
+class IntList(types.TypeDecorator[list[int]]):


I introduce a sqlalchemy custom type to encapsulate transformation of list of ints to comma separated list

sylvlecl · 2026-08-25T16:57:03Z

+    statistic_type: Mapped[str | None]
+
+
+class DbParquetArea(Base):


Important:
this is the source of truth for what variables an area contain

sylvlecl · 2026-08-25T16:58:10Z

+    mc_years: Mapped[list[int]] = mapped_column(IntList)
+
+
+class DbParquetVariable(Base):


Important:
this is the source of truth for what each column of parquet files represent.

the column names of parquet files should not be relied upon

sylvlecl · 2026-08-25T16:59:11Z

+MC_IND_AREA_COL_OFFSET = len(MC_IND_AREA_INDEX)
+
+
+def iterate_areas_df(


This function contains the complexity of retrieving only the right columns for the right areas

sylvlecl · 2026-08-25T17:01:03Z

+    variables: Sequence[int]
+
+
+class IParquetOutputMetadata(ABC):


This is aimed at being the central object used for retrieving information about the output.
Retrieval of other metadata could be moved here too.

sylvlecl · 2026-08-25T17:02:57Z

+    def get_variables(
+        self, aggregation: ScenarioAggregation, element_type: ElementType
+    ) -> Sequence[VariableDescription]:
+        return self.variables_index.get_variables(aggregation, element_type)


The assumption is that retrieving all variable definitions is OK for any use case.
Could be refactored if it becomes a problem, but it looks OK.

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl · 2026-08-25T17:19:07Z

+    raise NotImplementedError(f"Not yet implemented: {metadata.file_type}")
+
+
+def extract_areas_refacto(


To be renamed later, of course :)

sylvlecl · 2026-08-25T17:19:26Z

+    """
+    Creates parquet files in target_dir in consistence with columns that have been defined in the metadata object.
+    """
+    extract_areas_refacto(metadata, file_output, target_dir)


Other element types to be added

sylvlecl · 2026-08-25T17:32:56Z

+    data: pl.DataFrame
+
+
+class ParquetOutputWriter:


That class plays an important part too:

it allows to append to the parquet file dataframes that only have a subset of the desired columns, and in any order.

sylvlecl · 2026-08-25T17:33:49Z

+            self.writer.close()
+
+    def _create_schema(self) -> pa.Schema:
+        return pa.schema(


We now create explicitly the arrow schema, based on the columns definition.

sylvlecl · 2026-08-25T17:35:15Z

+    output = FileOutput(output_dir)
+    parsing_result = parse_output_variables(output)
+
+    # Notes: weird stuff in input data: CO2 emissions in MWh, and different naming for thermal production groups


For information, this has been fixed since on antares-simulator side

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

pull-request-size Bot added the size/L label Aug 20, 2026

github-code-quality Bot found potential problems Aug 20, 2026

View reviewed changes

Comment thread antarest/output/filestudy/model.py

from functools import cached_property

from pathlib import Path

from typing import Callable, Generic, Literal, Sequence, TypeAlias, TypeVar

from typing import Callable, Generic, Iterable, Literal, Sequence, TypeAlias, TypeVar

sylvlecl added 2 commits August 21, 2026 12:11

unit test for variables parsing

15d8257

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

fix mc-all parsing

2a355d9

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

pull-request-size Bot added size/XL and removed size/L labels Aug 21, 2026

sylvlecl added 6 commits August 21, 2026 14:47

make parsing fetch data from the right dir

46375b0

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

remove test, add mc-all files

0f46f49

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

move functions from dbmodel

fd259be

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

unit test for fetch from database

2b5e48d

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

add some todos, start re-working parquet extraction

56c3ffb

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

remove unused fixture dependency

fd371a7

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl commented Aug 21, 2026

View reviewed changes

add more comments

fdb781f

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

pull-request-size Bot added size/XXL and removed size/XL labels Aug 21, 2026

sylvlecl commented Aug 21, 2026

View reviewed changes

sylvlecl added 8 commits August 22, 2026 10:39

implement columns adaptation

8c17bf0

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

add support for index columns in writer

b2d121c

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

fix mypy related issue

fbddea8

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

first test for parquet file creation

611465e

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

first tests for area parquet file creation

26b3395

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

wip

2ca742c

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

reorganize metadata implementation

f72dd4c

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

renaming

c12e2e4

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

github-code-quality Bot found potential problems Aug 24, 2026

View reviewed changes

Comment thread antarest/output/storage/v2/download.py Fixed

sylvlecl added 5 commits August 25, 2026 08:52

moving stuff

0ef2eef

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

clarify functions chaining

b9d1c14

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

add call site in storage implementation

ac9c9e3

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

remove comment

9e02854

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

separate iteration logic and fix it

2fde291

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

github-code-quality Bot found potential problems Aug 25, 2026

View reviewed changes

add some test, fix download impl

143ee0c

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl commented Aug 25, 2026

View reviewed changes

sylvlecl added 4 commits August 25, 2026 19:06

remove unused classes

3bb9d3a

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

remove unused methods

a0b5119

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

move helper class

87ce1b8

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

remove obsolete unit test

abd9ae3

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

sylvlecl commented Aug 25, 2026

View reviewed changes

remove unused fixture

b9eed3d

Signed-off-by: Sylvain Leclerc <sylvain.leclerc@rte-france.com>

		raise OutputSubFolderNotFound(output_dir.name, "economy\|adequacy")


		class FileOutput:


		__tablename__ = "parquet_output"

		id: Mapped[int] = mapped_column(BigInteger, primary_key=True, autoincrement=True)

		mc_ind_vars: Mapped[list[int]] = mapped_column(Columns)


		# TODO: add tables for other element types: links, thermal clusters, etc


		simulation_range = _extract_simulation_range(dir_path)

		# TODO: first, extract variables metadata to database

		yield parse_output_file(data_file, start_col)


		def _extract_areas(

		from antarest.study.model import MatrixFrequency


		def _parquet_file_name(element_type: ElementType, frequency: MatrixFrequency) -> str:

		ScenarioAggregation: TypeAlias = Literal["mc-ind", "mc-all"]


		class IntList(types.TypeDecorator[list[int]]):

		statistic_type: Mapped[str \| None]


		class DbParquetArea(Base):

		mc_years: Mapped[list[int]] = mapped_column(IntList)


		class DbParquetVariable(Base):

		MC_IND_AREA_COL_OFFSET = len(MC_IND_AREA_INDEX)


		def iterate_areas_df(

		raise NotImplementedError(f"Not yet implemented: {metadata.file_type}")


		def extract_areas_refacto(

Uh oh!

Conversation

sylvlecl commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvlecl Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvlecl Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvlecl Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

sylvlecl commented Aug 20, 2026 •

edited

Loading

sylvlecl Aug 21, 2026 •

edited

Loading

sylvlecl Aug 21, 2026 •

edited

Loading

sylvlecl Aug 25, 2026 •

edited

Loading