Skip to content

feat(test-cli): add JSON output and optimization analysis to groupstats - #3308

Open
danceratopz wants to merge 1 commit into
ethereum:forks/amsterdamfrom
danceratopz:groupstats-agentic
Open

feat(test-cli): add JSON output and optimization analysis to groupstats#3308
danceratopz wants to merge 1 commit into
ethereum:forks/amsterdamfrom
danceratopz:groupstats-agentic

Conversation

@danceratopz

@danceratopz danceratopz commented Aug 5, 2026

Copy link
Copy Markdown
Member

Description

Improve the groupstats pre-allocation group analysis utility so it is useful for both human inspection and programmatic/agent-driven analysis.

The command now supports:

  • --output rich|json to switch between the existing rich console view and machine-readable JSON.
  • --low-test-count to define which groups should be treated as optimization targets.
  • --limit to bound candidate result lists, with 0 meaning no limit.
  • --include-test-ids to opt into full test ID lists.
  • --include-group-details/--exclude-group-details to control per-group output size.
  • --compact to omit verbose top-level maps while keeping totals, distributions, fork stats, and optimization candidates.
  • --match-test-id and --match-test-id-regex to focus analysis on matching tests before recomputing rich or JSON stats.
  • --exclude-test-id and --exclude-test-id-regex to remove known-noisy tests before recomputing rich or JSON stats.

The command help now includes an agent/programmatic example and points callers at the JSON optimization object, especially candidate_buckets, as the first place to inspect for possible larger group opportunities.

The analyzer now reads pre-alloc group JSON directly into lightweight records instead of loading full fixture models, and reports ranked optimization data for:

  • low-count groups,
  • candidate buckets sharing fork, chain ID, group salt, and environment,
  • test-function candidates,
  • module candidates.

The rich output keeps the existing at-a-glance tables and adds singleton/low-count summaries plus a candidate-bucket table.

Validated via:

  • uv run pytest packages/testing/src/execution_testing/cli/tests/test_show_pre_alloc_group_stats.py
  • just static
  • uv run groupstats --help
  • Manual validation against the cached tests%40v20.0.1 blockchain_tests_engine_x/pre_alloc fixture directory in both rich and JSON modes.

Related Issues or PRs

N/A.

Checklist

  • Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
  • PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

[A judgemental cat in a box, inspecting your pre-alloc groups

…ats`

Make the pre-alloc group statistics utility useful for both human
inspection and programmatic/agent-driven analysis:

- Add `--output rich|json` to switch between the existing rich console
  view and machine-readable JSON.
- Add `--low-test-count`, `--limit`, `--include-test-ids`,
  `--include-group-details/--exclude-group-details`, and `--compact`
  to define optimization targets and bound output size.
- Add `--match-test-id[-regex]` and `--exclude-test-id[-regex]` to
  focus the analysis or drop known-noisy tests before recomputing
  stats.
- Read pre-alloc group JSON directly into lightweight records instead
  of loading full fixture models.
- Report ranked optimization data (low-count groups, candidate buckets
  sharing fork, chain ID, group salt, and environment, test-function
  and module candidates) in JSON, and add singleton/low-count summaries
  plus a candidate-bucket table to the rich output.
@danceratopz danceratopz added A-test-cli Area: execution_testing.cli C-feat Category: an improvement or new feature labels Aug 5, 2026
@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.49%. Comparing base (5bb2a51) to head (25c507b).

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #3308   +/-   ##
================================================
  Coverage            93.49%   93.49%           
================================================
  Files                  624      624           
  Lines                37056    37056           
  Branches              3394     3394           
================================================
  Hits                 34647    34647           
  Misses                1653     1653           
  Partials               756      756           
Flag Coverage Δ
unittests 93.49% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-cli Area: execution_testing.cli C-feat Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant