Skip to content

refactor: refactor statistics computation for faster stats on large datasets - #172

Merged
vrahier merged 4 commits into
mainfrom
refactor/stats-pairs
Oct 6, 2026
Merged

vrahier merged 4 commits into
mainfrom
refactor/stats-pairs

Conversation

@vrahier

@vrahier vrahier commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR refactors statistics_situations() and statistics() in R/generic_stats.R. They now compute all statistics in one vectorised pass instead of looping over versions and situations. The exported API stays the same.

Motivation

The previous implementation called statistics() once for every version × situation pair. Each of those calls ran:

  • one dplyr::summarise() per statistic (around 38 with stat = "all");
  • a plyr::join_all() to merge the per-statistic results.

With many situations, this meant tens of thousands of small dplyr calls and joins. Their fixed overhead dominated the execution time.

Changes

  • New stats_pairs() builds one long table of observed/simulated pairs (group, situation, Plant, variable, Observed, Simulated) for all versions and situations. format_cropr() now runs once per situation, with all versions stacked together.
  • New situation_pairs() holds the per-situation formatting logic that used to live in statistics().
  • New compute_stats() computes every requested statistic in a single group_by() + summarise(). Groups and situations keep their order of appearance.
  • New stats_description() and check_stat_names() pull the statistic descriptions and the name validation out of statistics().
  • statistics() is now a thin wrapper around situation_pairs() and compute_stats().
  • cat_situations() / add_situation_col() are no longer used here. all_situations = TRUE now simply relabels the situation column.
  • The plyr dependency is removed.

@vrahier vrahier self-assigned this Oct 5, 2026
@vrahier vrahier changed the title refactor: improve statistics refactor: refactor statistics computation for faster stats on large datasets Oct 6, 2026
@vrahier
vrahier merged commit 8533aa2 into main Oct 6, 2026
8 of 10 checks passed
@vrahier
vrahier deleted the refactor/stats-pairs branch October 6, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant