Benchmark Full Suite Generation Tracker

[LouisTsai-Csie]

Description

To finalize repricing numbers, we need to run the complete benchmark test suite on both compute and stateful benchmarks (not repricing-specific).

Initial results on Benchmarkoor:

• state-actor-glamsterdam-devnet-7-full-stateful
• state-actor-glamsterdam-devnet-7-full-compute

This tracker documents what's missing or broken in the output.

Compute

Statistics: passed 2040, failed 204, skipped 22, deselected 2.

These scenarios are not constructed successfully via the stateful filler. The constructed blocks are not accepted by clients:

• test_ether_transfers_to_precompile
• test_ether_transfers: diff_acc* ignored (66/120)
• test_auth_transaction: empty_authority_False ignored (16/32)
• test_storage_access_cold: absent_slots_False ignored (14/28)
• test_selfdestruct_existing
• test_ext_account_query_cold: absent_accounts_False (2/4)
• test_mixed_dependency_graph: all max_txs + half of half_max_txs (10/18)
• test_deploy_then_interact: max_txs-single_contract (2/12)

Refactor PR: issue #3282

Below tests needs further refactor, the current storage initialization approach would exceed the initcode prefix size limit.

• test_state_root_computation: initcode prefix too long

Below tests are being skipped:

• test_unchunkified_bytecode: contract deployment gas exceeds the benchmark gas value
• test_creates_collisions: Benchmark gas value cannot cover a single contract creation
• test_bls12_pairing_uncachable[num_pairs_24]: calldata intrinsic gas cost exceeds the gas limit, this could be refactored.
• test_block_full_access_list_and_data: EIP-7934 block RLP size limit might be exceeded -> this expected but worth refactoring

Stateful

The test is limited to tests/benchmark/stateful/bloatnet, below is the failing cases:

1. depth_benchmarks/test_deep_branch.py
2. eip7928_block_level_access_lists/: not within the current scope.
3. test_account_query.py: test_balance_query, test_extcodesize_bytecode_sizes
4. test_sload.py: test_sload_bloated_prefetch_miss, test_sload_bloated_multi_contract[existing_slots_True]
5. test_sstore.py: test_sstore_dirty_transitions[oscillation_4x_from_zero]
6. test_transaction_types.py: test_ether_transfers_onchain_receivers[to_self]
7. test_erc20.py
8. test_account_access:non-existing scenario exceeds block gas limit.
9. test_ether_transfers_onchain_receivers: diff_to_nonexistent is not the worst case version, diff_to_self is broken due to RPC issue.

Suggestion to the next step:

1. Exclude p1, p2 and test_sload_bloated_multi_contract from this run, as this benchmark might require additional pre-state.
2. Remove tests in p3 and p7, since they are mostly covered by existing scenarios.

Comment here for visibility: #3106 (comment), now waiting for CPerezz' response.

This means we could focus on the following scenaio only:

• test_sload_bloated_prefetch_miss
• test_sstore_dirty_transitions[oscillation_4x_from_zero]
• test_ether_transfers_onchain_receivers[to_self]: *to_self costs only intrinsic gas per tx (12,000/tx), so a 300M gas target produces 25,000 transactions. build_block() RLP-encodes all of them into a single testing_buildBlockV1 request in one shot (it does not go through get_alloc's batching path) roughly 5.4-5.7MB total, which exceeds go-ethereum's plain HTTP RPC endpoint's hardcoded 5MB request-body limit (no CLI flag to raise it). At 60M the same case only produces 5,000 txs (~1.1MB), safely under the limit the 5x jump to 300M is what pushes it over.
• test_account_access - refactor(test-benchmark): split stateful setup transactions across blocks #3282
• test_ether_transfers_onchain_receivers - refactor(test-benchmark): split stateful setup transactions across blocks #3282

Notes

• Backports feat(test-benchmark): add missing glam eip to benchmark #3267 to benchmarks/amsterdam -> Done by PR feat(test-benchmark): add missing glam eip to benchmark [backport benchmarks/amsterdam] #3276
• Backports feat(test-benchmark): enhance worst case coverage #3274 to benchmarks/amsterdam -> Done by PR feat(test-benchmark): enhance worst case coverage [backport forks/amsterdam] #3277

[qu0b]

One request to fold into the benchmark refactors: could the overhead_baseline calibration variants (and any future helper-only cases) declare their role in the fixture output — e.g. a benchmark_role: calibration field in the fixture _info/metadata — rather than only in the test-name params?

Context: benchmarkoor now excludes overhead_baseline_True variants from its statistics by default (ethpandaops/benchmarkoor#293, per the #benchmarking discussion — they stay filled and executed, analysis like Maria's subtraction still gets the raw rows). The exclusion currently has to match on the test-name substring, which silently breaks if the param is ever renamed and can't distinguish future helper families. A machine-readable role in the fixture metadata would let consumers classify without name conventions.