feat(builder): add builder.pre_runs stage (gas-bump + funding + setup fill)#285
Open
skylenet wants to merge 22 commits into
Open
feat(builder): add builder.pre_runs stage (gas-bump + funding + setup fill)#285skylenet wants to merge 22 commits into
skylenet wants to merge 22 commits into
Conversation
… fill) Adds an optional builder stage that runs between state_actor and eest_payloads. Per target it copies a snapshot datadir into output_dir, boots a filler EL client on it, ramps the block gas limit to a target (default 1 TGas) via empty blocks, builds a funding block, then runs fill-stateful with --no-reset-between-tests on configured setup tests so the deployed state persists on the datadir. The filler is stopped and output_dir holds the advanced datadir, which a later eest_payloads target uses as its source_dir — so the benchmark fill builds on top of the setup (e.g. bloatnet test_account_access recomputes the deterministic CREATE2 addresses that test_setup_contracts deployed; no manifest is exchanged). Gas-bump and funding blocks are built via the Engine API (testing_buildBlockV1 -> engine_newPayload -> engine_forkchoiceUpdatedV3, JWT-signed), mirroring NethermindEth/gas-benchmarks. The gas bump loops until the head gas limit reaches the target (self-terminating, with a safety cap) rather than a fixed block count. Requires fill-stateful's --no-reset-between-tests flag (skylenet/ execution-specs feat/fill-stateful-no-reset-between-tests). The filler must implement testing_buildBlockV1, so the same client set as eest_payloads applies: geth, besu, nethermind. Adds config.state-actor-eest.simple.amsterdam.stateful.pre-runs.yaml as a worked example.
Two fixes found by an end-to-end smoke run of builder.pre_runs, plus a new fill_env knob: - testing_buildBlockV1 lives in the `testing` namespace, exposed on the filler's unauthenticated HTTP RPC port — not the authrpc/engine port (geth's authrpc does not serve it). Call it on the RPC URL without JWT; only newPayload/forkchoiceUpdated go to the engine port with JWT. Previously every gas-bump/funding block failed with "testing_buildBlockV1 does not exist/is not available". - Add builder.pre_runs fill_env: extra environment variables for the fill-stateful container (e.g. BLOATNET_RECEIVER_CONTRACT_COUNT to shrink the setup for a smoke run). Viper lowercases map keys, which would break case-sensitive env-var names, so restore their original casing from the raw YAML (mirrors restoreEnvironmentKeyCasing / address_stubs). Verified end-to-end against a geth filler on a state-actor snapshot: the gas-bump, funding block, and fill-stateful invocation (correct flags + env) all succeed; the setup blocks build on the live filler.
The skylenet/geth:bal-devnet-7 build caps contract code at 32KB, but the EEST library's Amsterdam.max_code_size() is 64KB, so the bloatnet setup's 64KB 'max' contracts revert with 'max code size exceeded'. Switch the geth filler/runner image to ethpandaops/geth:glamsterdam-devnet-6 (64KB limit), validated end-to-end, and note that the besu/nethermind filler images must be matching glamsterdam-devnet-6 builds for the same limit.
Point the besu/nethermind fillers (and runner instances) at ethpandaops/<client>:glamsterdam-devnet-6 to match geth, and extend nethermind's genesis_eip_override with the glamsterdam-devnet-6 EIPs (2780, 7997, 8038, 8246, 8282) on top of the existing set. geth/besu activate the whole amsterdam fork; only nethermind needs the per-EIP list. geth is validated end-to-end; besu/nethermind are set but unverified.
Adds a replay path so clients that cannot act as the fill-stateful filler (reth/ethrex) still get the pre-run datadir. A builder target now records a replayable bundle of its gas-bump/funding/setup engine_newPayloads (bump/funding captured as built; setup blocks extracted from the fixtures), written to <output_dir>/.pre-run-payloads.jsonl. A target with `replay_from: <builder>` boots its client on its own snapshot (via the client spec's standard launch command) and replays that bundle block-by- block, then verifies it landed on the builder's head. Validated end-to-end geth->geth: the replay client reproduces the builder's exact head and state root from a 14-payload bundle. Recording only happens when a replay target references the builder, so plain builders pay nothing. erigon replay (needs a genesis-import init container) is guarded with a clear error; the example config wires reth/ethrex replay targets.
…ients Two fixes found by a multi-client replay run (geth builder -> reth/nethermind replay): - Verify the replay landed on the builder's head by querying the block by NUMBER, not "latest". nethermind leaves its `latest` pointer at the snapshot head after newPayload/forkchoiceUpdated even though the block was applied, so the old "latest" check spuriously failed there. - Point HOME/XDG_CACHE_HOME at a writable dir for the replay client. Running as the host user, reth otherwise tries to create its cache/log dir under / and exits with "permission denied" before its RPC comes up. Validated: geth (filler) + reth (replay) both boot on their pre-run datadirs (same head) and run the geth-filled test_account_access fixtures, 10/10 each.
…ld time
Replaces the build-time replay_from mechanism with a run-time one. A single
filler pre_run target (geth) now records a replayable payload bundle
(gas-bump + funding + setup engine_newPayloads) as ".request" lines under
output_dir/pre_run_bundle/. The runner replays that bundle once per instance,
before the benchmark fixtures, as a session-level pre-run step
(eest_fixtures.pre_runs.{local_fixtures_dir,fixtures_subdir}). Every client
boots on its own RAW snapshot; SkipUntilBlockNumber makes the bundle a no-op
when a datadir is already advanced.
Why: the build-time replay had to re-implement each client's boot
(startReplayClient), which broke on besu/nethermind/ethrex/erigon. The runner
already launches every client correctly and its runStepLines has SYNCING/retry
handling, so replaying there "just works" — no per-client advanced datadirs.
Removed: PreRunTarget.replay_from + IsReplay + replay validation;
PreRunsBuilder.runReplay/startReplayClient; engineClient.replayPayloads and the
build-time head-check helpers. Bundle format is now newPayload+forkchoiceUpdated
request lines the runner consumes directly.
Validated end-to-end: geth (filler) and reth (non-filler, advanced solely by the
runner replaying geth's bundle on reth's raw snapshot) both pass 10/10 with 0
INVALID / 0 SYNCING. nethermind cross-client re-execution still INVALIDs (a
per-client snapshot-state issue, now in the runner replay path) — follow-up.
Strict clients (besu, nethermind, ethrex) reject every block once amsterdam is active unless the two EIP-8282 builder-execution-request predeploys have code — they are system-called at the end of each block. The osaka state-actor snapshot didn't carry them, so besu/ethrex INVALIDed the pre-run bundle's very first block (geth/reth tolerated the absence). Predeploy both with the ethereum-genesis-generator v6.1.2 (glamsterdam-devnet-6) bytecode in state_actor.spec.entities: 0x0000884d2AA32eAa155F59A2f24eFa73D9008282 (deposit requests) 0x000014574A74c805590AFF9499fc7A690f008282 (exit requests) Code-only is enough: the end-of-block system call reads the request queue (empty), so it returns empty requests and validates — matching geth. The slot-0 sentinel only matters for user request submissions, which these blocks don't make (state-actor's spec has no storage field anyway). Validated: geth/besu/reth/ethrex all run the pre-run bundle + fixtures with 0 INVALID (besu and ethrex go from 0/10 to 10/10; reth 9/10 is an unrelated flake).
…h commented Add nethermind as the pre_runs + eest_payloads filler in the amsterdam stateful pre-runs example config and make the runner source selectable via EEST_FIXTURES_RUNNER_SOURCE (default nethermind). The geth filler is kept commented out as an alternative. Validated end-to-end: nethermind builds its pre-run bundle + fixtures and the runner replays them green on geth and nethermind. Also fix two issues found validating the config end-to-end: - eest_payloads filter now excludes the 5 test_account_access CALL-with-value variants that revert (receipt status 0) when filled on the setup datadir and would otherwise fail the whole build; keeps the other 95. - Drop besu --Xbal-processing-timeout / --Xbal-state-root-timeout: the glamsterdam-devnet-6 besu image rejects them (exit code 2, "Unknown options"). Update TestLoad_PreRunsExampleConfig for the single active (nethermind) pre-run target.
Activate fill_env.BLOATNET_RECEIVER_CONTRACT_COUNT=1000 in the amsterdam stateful pre-runs example so the pre-run's test_setup_contracts fill deploys 1000 receivers per mode instead of the 100k default, keeping the setup fill tractable. Raise or remove for a full-scale run.
…bundle A pre_runs target with replay_from skips the fill and instead boots its client on a copy of its snapshot and replays a recorded .request bundle onto it (engine API, asserting each payload VALID), so output_dir becomes the advanced datadir. This advances each client at build time — one client fills, the rest replay its bundle — so the runner boots every client from its own advanced datadir with no per-test run-time replay (which is slow for a large pre-run). replay_from resolves two ways (single field): a declared non-replay pre_runs target name (-> its output_dir/pre_run_bundle bundle) or an absolute path to a .request file / pre_run_bundle directory. It works for non-filler clients (reth/ethrex) — the case the removed build-time replay could not boot — by reusing the client-agnostic startFiller with the client's standard runner command (spec.DefaultCommand) instead of the geth-style fill command, a writable HOME (reth writes ~/.cache), and extra_args that override matching default flags (besu rejects a duplicate --p2p-enabled). Validated: nethermind fills; reth (non-filler) and besu advance purely by replaying nethermind's bundle, and reth then runs the benchmark from its replay-advanced datadir (boots at the setup head, 0 INVALID). - config: PreRunTarget.replay_from + IsReplay + validateReplayFrom. - builder: extracted shared bootFiller; runReplay + resolveReplayBundle; engineClient.replayBundle (SYNCING retry); fillerReplayCommand + fillerBootEnv + overrideArgs; startFiller takes the boot command. - example config switched to the build-time layout (nethermind fill + geth/besu/reth/ethrex replay_from; runner boots each from its advanced datadir).
eest_payloads fills its fixtures on the pre_runs advanced datadir, so a rebuilt pre-run leaves any existing eest fixtures stale. runBuilders now tracks whether any builder.pre_runs target actually built (not skipped) and, since targets run in declaration order (pre_runs before eest_payloads), forces the eest_payloads targets to rebuild even when their output_dir looks populated. Tested via TestRunBuilders_ForceEESTAfterPreRuns (fakeBuilder now records the Force option): eest is forced after a pre_runs rebuild, not forced when pre_runs was skipped.
…g snapshots Advance an OSAKA snapshot to AMSTERDAM at build time, deploying the target fork's system contracts (e.g. the EIP-8282 builder-execution-request contracts) before it activates — a strict client rejects post-fork blocks whose system contracts have no code. - engineClient crosses a fork boundary per block: blocks timestamped before the activation TS build as the pre-fork (newPayloadV4, no slotNumber), at/after as the target fork (V5). buildBlock now carries transactions; deployContracts signs a contract-creation tx per runtime, imports one pre-fork block, and verifies each CREATE address received code. - config: `predeploy` on a pre_runs target (pre_fork, deployer_key, contracts); amsterdam activation reuses genesis_eip_override.timestamp. Full validation. - pre_runs with datadir_method: schelk now runs `schelk restore` (recover + mount) on the source before advancing, so it starts from a clean baseline. Kept out of the shared source-mount path, so later stages only mount and can never wipe the advanced volume. No promote anywhere. - example config.existing-snapshot-eest...pre-runs.yaml: osaka snapshot → deploy 8282 on osaka → cross to amsterdam → fill; eest_payloads and runner boot direct on the pre-run output (no copy). nethermind hardcodes the 8282 addresses (not a chainspec param), so the deployed contracts land at the deployer's CREATE addresses; the amsterdam stages still need a nethermind build that relaxes the fixed-address system-call check.
…st_payloads The runner already downloaded genesis URLs; the builder read genesis only as a local file (stat/read/bind-mount/sha256) and config validation required an absolute path. Add matching URL support so a chainspec can be fetched at build time instead of staged locally. - resolveGenesisFile downloads an http(s) ref to a temp file (0644, container- mountable) with a cleanup, and passes local paths through unchanged. Wired into both builders right after the fast-path skip, so a populated/unforced target never fetches; everything downstream works on the resolved path. For eest the fingerprint hashes the downloaded content, so a changed URL triggers a rebuild. - eest_payloads genesis validation now accepts an absolute path OR an http(s) URL (pre_runs never had the abs-path check). - example config.existing-snapshot-eest...: NETHERMIND_CHAINSPEC points at the devnet-3 osaka chainspec URL. Also bump both *-pre-runs.yaml examples to the LouisTsai-Csie/execution-specs full-benchmark-target EEST ref.
"RETURNs"/"CODECOPYs" tokenize as RETUR+Ns / CODECOP+Ys, and typos flags RETUR as a misspelling of RETURN. Phrase the opcodes as CODECOPY/RETURN in parens instead of appending a lowercase plural.
Exercise the pre_runs builder end-to-end in CI: state-actor builds a 256 MB OSAKA besu snapshot, pre_runs advances it (a tiny test_setup_contracts fill via the besu filler, recording a replay bundle), eest_payloads fills the bloatnet account-access benchmark on that advanced datadir, then the runner replays those fixtures on besu. Kept as its own job rather than folded into state-actor: that job fills the bn128 compute subset (no state built), whereas pre_runs is an advance-then- benchmark flow, so merging would drop the compute coverage. Single-besu + osaka keeps it within a GitHub-hosted runner and needs no amsterdam system-contract predeploys.
The build-artifact archive step grouped targets by builder but only handled eest-payloads (full fixtures) and state-actor (metadata only) — pre-runs targets fell through and contributed nothing. Add a pre-runs branch that stages just the replay bundle (pre_run_bundle/pre-run.request), never the multi-GB advanced datadir. Guarded so replay-only targets (which have no bundle) are skipped.
The pre-runs and state-actor jobs both upload build artifacts with the default name (benchmarkoor-build-<run_id>), producing two same-named artifacts on one run — ambiguous to download. Name the pre-runs one benchmarkoor-build-pre-runs-<run_id>.
bootFiller always copied source_dir → output_dir before booting, which cannot work for a multi-TB snapshot (no disk holds a full copy). For datadir_method: schelk the source is a copy-on-write scratch, so boot the filler directly on source_dir (after recover+mount) — no copy — and treat source_dir as the advanced datadir. output_dir is unused for schelk: it is no longer required, IsInPlace()/ AdvancedDir() report source_dir, and the replay bundle is written there. A schelk target always runs (its scratch is always populated), restoring to a clean baseline first. The example existing-snapshot config now points every stage (pre_runs source, eest_payloads source, runner datadir) at the schelk snapshot path and drops the /tmp PRERUN_DIR, so the whole pipeline operates in place on the CoW scratch.
…ot example Learnings from running the osaka→amsterdam pre-run on a real 1.4 TB ethpandaops mainnet nethermind snapshot: - source_dir must be the CHAIN column dir (…/nethermind_db/<chain>, e.g. …/nethermind_db/mainnet) — nethermind reads columns directly under --Init.BaseDbPath, so pointing at …/nethermind_db boots at genesis. - real nethermind snapshots store state in FlatDb; add --FlatDb.Enabled=true (+ the compaction/robustness flags the snapshot's node used) via filler_extra_args on both stages, else nethermind throws MissingTrieNodeException on boot. - in-place (schelk) boots on the snapshot owned by its fetcher UID, so run `benchmarkoor build` as root until the filler can run as the datadir's owner. - the amsterdam activation ts must exceed the ACTUAL head block ts (from the live client), not the possibly-stale _snapshot_*.json block.
Stateful benchmarks that draw senders from EEST's yield_distinct_sender pool
(e.g. test_ether_transfers_onchain_receivers) fail on a snapshot where the pool
is unfunded ("expected N transactions but only 0 were included"). Add a
funding_pools config field that credits a derived pool in the pre-run funding
block: address i is the EOA of key int(keccak256(base_key_seed)) + i, matching
execution-specs' SENDER_BASE_KEY = int(keccak256("gas-repricings-private-key")).
deriveSenderPool generates the addresses; expandFundingAccounts folds each pool
into the funding-block withdrawals alongside explicit funding_accounts. Beats
hand-listing hundreds of addresses (833 for the 10 Mgas ether-transfer variant).
…nt gas-model failures Raise eest_payloads gas_benchmark_values to 300 Mgas and the yield_distinct_sender funding pool to 25000 (the largest per-variant sender fan-out at that budget, so txs aren't dropped as "expected N but only <count> included"). Broaden the fill to the full bloatnet repricing suite with the bloated_eoa_10GB address stub, and add a filter excluding the 11 value-to-contract variants that run out of gas on nethermind glamsterdam-devnet-6 (EEST's per-op gas model under-predicts the client's charge for value-bearing calls to contract accounts). See docs/eest-300m-known-failures.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional
builder.pre_runsstage that runs betweenstate_actorandeest_payloads. It advances a snapshot datadir so the benchmark fill can build on top of pre-deployed setup state (e.g. the bloatnettest_setup_contracts→test_account_accessflow).Per target it:
output_dirand boots a filler EL client on it in place.gas_limit(default 1 TGas), self-terminating with a safety cap.funding_accountsvia beacon withdrawals.fill-stateful --no-reset-between-testson the configuredtests, so the deployed state persists on the datadir instead of being rewound.output_dirnow holds the advanced datadir, which a latereest_payloadstarget consumes as itssource_dir.The setup→benchmark dependency is via deterministic CREATE2 (the query test recomputes the addresses the setup deployed), so no manifest is exchanged between the two fills.
How
testing_buildBlockV1→engine_newPayloadV{4,5}→engine_forkchoiceUpdatedV3, JWT (HS256) signed — mirroringNethermindEth/gas-benchmarks'eest_stateful_generator.py(pkg/builder/engine.go).pkg/builder/pre_runs.go) reusesEESTPayloadsBuilder's filler-boot and fill-image machinery; it boots the filler onoutput_dirwith datadir methoddirectso the filler's writes are the persisted output.PreRunsConfig/PreRunTargetmirror the eest_payloads shape (hoistableconfigdefaults,Resolve*, validation, JWT default) (pkg/config/pre_runs.go), wired intoBuilderConfigand thebuildcommand (runs afterstate_actor, beforeeest_payloads; adds--skip-pre-runs-build/--limit-pre-runs-target).Dependency
Requires fill-stateful's
--no-reset-between-testsflag: skylenet/execution-specsfeat/fill-stateful-no-reset-between-tests(branched offLouisTsai-Csie/execution-specsfull-benchmark-target, so it also carries the bloatnet tests). The example config pointseest_repo/eest_refat it.Filler-client constraint
The filler must implement
testing_buildBlockV1, sopre_runssupports the same clients aseest_payloads: geth, besu, nethermind. reth/ethrex/erigon can still be benchmarked on the resulting datadir, but cannot produce the pre-run datadir themselves.Testing
go build ./...,go vet, andgolangci-lint --new-from-rev=origin/masterare clean; builder + config + cmd tests pass (pre-existing macOS/proc/mountsschelk tests excepted).Adds
examples/configuration/config.state-actor-eest.simple.amsterdam.stateful.pre-runs.yaml.