fix(spec-tools,testing): move evm_tools into the testing package

[gurukamath]

Description

pip install-ing this repository produces a broken ethereum-spec-evm: the t8n
and statetest subcommands import execution_testing (since #2924), which is a
uv workspace member — not a dependency of ethereum-execution — so standalone
installs fail with ModuleNotFoundError on first use.

This PR fixes the dependency inversion by moving evm_tools (t8n, b11r,
statetest, daemon) into the ethereum-execution-testing package, so all
imports point one way: testing → spec. The ethereum-spec-evm entry point
moves with it; loaders/ and utils.py stay in ethereum_spec_tools.

• CI: new job builds all workspace wheels, installs them by explicit path
  into a clean venv, and runs a real Frontier transition — the class of
  regression that shipped No module named 'execution_testing' #3236 is now caught before merge.
• Docs: new packages/testing/README.md (also fixes its dangling
  readme = metadata) with verified standalone install recipes, plus updated
  repo README, packaging docs, and an evm_tools reference page.

Note for consumers: the CLI is no longer installable from the repo root alone —
install both packages from the same clone, e.g.
pip install ./execution-specs ./execution-specs/packages/testing.

Related Issues or PRs

Fixes #3236.

Checklist

• Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
• PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.50%. Comparing base (5bb2a51) to head (1beeb41).
⚠️ Report is 6 commits behind head on forks/amsterdam.

Additional details and impacted files

@@               Coverage Diff                @@
##           forks/amsterdam    #3307   +/-   ##
================================================
  Coverage            93.49%   93.50%           
================================================
  Files                  624      624           
  Lines                37056    37070   +14     
  Branches              3394     3394           
================================================
+ Hits                 34647    34661   +14     
  Misses                1653     1653           
  Partials               756      756           

Flag	Coverage Δ
unittests	93.50% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[spencer-tb]

One small thing (the diff view hides renamed file). The Evm docstring in evm_trace/protocols.py cross references the old module path, and now that the file lives in the testing package it is rendered by mkdocstrings rather than docc, here

A larger thing, can we remove b11r? I remember there was talk of security using it for fuzzing but I think they should or infact do use geth's now :)

[spencer-tb]

Should we move tests/evm_tools somewhere else in the repo? I think we should try to aim for src/ only including the spec and tests/ only including the spec/bench tests

Definitely follow ups!

[gurukamath]

Yes. The folder name is a bit misleading. Happy to fix in a follow up

[danceratopz]

A quick win for clarity, potentially in this PR: rename tests/evm_tools to tests/spec_tools, matching the just recipe name (spec-tools). After the move the directory only holds spec-tools tests anyway (test_lint.py, test_new_fork.py, test_docc_shards.py).

The rename touches three places: the recipe at Justfile:219, the fill default ignores at packages/testing/src/execution_testing/cli/pytest_commands/fill.py:143, and the three t8n_build paths in the packaging job. If the t8n_build inputs move to the testing package (see the comment above on the workflow), the last one disappears and this becomes a git mv plus two lines.

[gurukamath]

One small thing (the diff view hides renamed file). The Evm docstring in evm_trace/protocols.py cross references the old module path

Fixed the docstrings. Nice catch!!

A larger thing, can we remove b11r? I remember there was talk of security using it for fuzzing but I think they should or infact do use geth's now :)

We could do this but I'd defer to a follow-up PR

[spencer-tb]

LGTM! :D

[danceratopz]

Thanks @gurukamath, the move looks clean and (bar 2. below) complete!!!

Details are in the inline comments. Two larger changes that could/should go in this PR:

1. Move the new packaging job into a just test-packaging recipe (see the comment on the job).
2. Let loaders/ and utils.py follow evm_tools into the testing package (see the comment on transaction_loader.py). This PR already breaks their import path, so one break beats two.

Follow-up: opened #3326 for the unrelated but never-runnable test_count_opcodes.py that was spotted during review.

[danceratopz]

My friend and I tested this step against the pre-fix wheels: built the forks/amsterdam spec wheel, installed it alone into a clean venv, and ran this exact walk-import script. It passes (exit 0) on the broken code.

The reason: the old execution_testing imports in evm_tools were all function-scoped (t8n/cli.py:214, :266, :314-322), deliberately, to avoid an import cycle. So every module imports cleanly, and #3236 only fired at runtime inside build_t8n_from_cli_options. That is exactly the traceback in the issue. A module-level import walk cannot see this form.

The step is still worth keeping. It catches future module-level leaks (now the likely form, since the cycle that motivated function-scoping is gone) and general wheel breakage.

To ensure the invariant "./src/ never imports execution_testing" does not get broken in the future, we could add an ethereum-spec-lint rule that rejects any such import, top-level or function-scoped. An AST check sees both forms, and it runs locally via just static. That might be a bit heavy though, so a quick alternative is ! grep -rn "execution_testing" src/, which passes as of this branch.

[danceratopz]

Now that t8n lives in the testing package, I think these new fixtures should be there too?

Suggested home: packages/testing/src/execution_testing/evm_tools/tests/fixtures/t8n_build/. This follows the existing convention in the package (client_clis/tests/fixtures/, specs/tests/fixtures/, tools/tests/test_filling/fixtures/).

These three lines are the only references repo-wide. The exclude = ["*tests*"] rule keeps the files out of the wheel, and this job reads them from the checkout, so only the paths here change.

It would also leave tests/evm_tools/ holding nothing but spec-tools tests, which makes the tests/spec_tools/ rename follow-up a pure git mv.

(One caution: not the top-level execution_testing/fixtures/, which is the fixture-format code subpackage.)

[danceratopz]

Justification about the move/rename of ./tests/evm_tools/ to ./tests/spec_tools/ here #3307 (comment)

[danceratopz]

The import-cycle note that lived here was deleted, but two comments in this file still point at it: lines 212 and 264 both read # Function-scoped: see import-cycle note in build_t8n_from_cli_options.

They now reference nothing, and these bare function-scoped imports invite a future hoist-to-module-level cleanup.

I would either restore a short note here (updated for the new home) or repoint the two comments. result.py:24-28 kept and updated its equivalent note in this PR and reads well as the model.

[danceratopz]

A quick win for clarity, potentially in this PR: rename tests/evm_tools to tests/spec_tools, matching the just recipe name (spec-tools). After the move the directory only holds spec-tools tests anyway (test_lint.py, test_new_fork.py, test_docc_shards.py).

The rename touches three places: the recipe at Justfile:219, the fill default ignores at packages/testing/src/execution_testing/cli/pytest_commands/fill.py:143, and the three t8n_build paths in the packaging job. If the t8n_build inputs move to the testing package (see the comment above on the workflow), the last one disappears and this becomes a git mv plus two lines.

[danceratopz]

This entry point is broken in a standalone wheel install: ethereum_spec_tools.whitelist is not in the [tool.setuptools] packages list, so it never ships and the script fails with ModuleNotFoundError. This is the same class of packaging bug as #3236.

What do you think about removing this line. It is a repo-bound dev tool (it edits the checkout's whitelist.txt) and its generic whitelist name lands in every consumer venv's bin/. The only consumer is Justfile:67, which can become uv run python -m ethereum_spec_tools.whitelist "$@". just whitelist <word> stays the same.

[danceratopz]

Can we permalink/SHA pin this? This points at tree/forks/amsterdam/..., which will break at the next fork rollover when the default branch moves on.

[danceratopz]

Now that t8n lives in the testing package, I think these new fixtures should be there too? See
https://github.com/ethereum/execution-specs/pull/3307/changes#r3725869393

[danceratopz]

I don't think we need a reference to the archived EEST repo here.

Suggested change

	Test generation and execution framework for the [Ethereum Execution Layer Specifications (EELS)](https://github.com/ethereum/execution-specs), derived from [ethereum/execution-spec-tests](https://github.com/ethereum/execution-spec-tests).
	Test generation and execution framework for the [Ethereum Execution Layer Specifications (EELS)](https://github.com/ethereum/execution-specs).

[danceratopz]

Thanks a lot for adding these additional checks, Guru!

Can we plz also add easy local verification of these steps by adding these checks to the Justfile instead? In particular, workflow-yaml-inline scripts make it hard to verify locally.

Every other test job here is a one-liner delegating to a just recipe; this is the only one with inline logic, including ~18 lines of Python in a heredoc that no linter or type checker ever sees. Moving the whole job into a just test-packaging recipe would match the other nine jobs, make the checks runnable locally before pushing (#3236 is exactly the class of bug you want to catch at the desk), and give the embedded Python a lintable home in .github/scripts/, which already has a tested-scripts convention via just test-ci-scripts.

Suggested shape:

# --- Packaging ---

# Build every workspace wheel into .just/dist
[group('packaging')]
build-wheels:
    uv build --wheel --all-packages --out-dir "{{ output_dir }}/dist"

# Smoke-test the built wheels: clean-venv install, real t8n run, spec-wheel-alone import check
[group('packaging')]
test-packaging: build-wheels
    # 1. install both wheels by explicit path into a fresh venv
    # 2. run the Frontier t8n transition, assert result.json
    # 3. install the spec wheel alone, run .github/scripts/import_check.py

The packaging group is forward looking: a future publish-wheels or a version-lockstep check would land in the same group, each independently runnable and clustered together in just --list. Note the doc comment above each recipe must be a single line; just shows only the line immediately above the recipe in the listing.

The job then reduces to checkout, setup-uv, run: just test-packaging (plus needs: static, see the other comment). One tradeoff: the four named steps collapse into one, so per-step timing and failure attribution moves from the UI into the log. The step comments here (why --all-packages, why explicit wheel paths, why a real transition instead of --help) are good and should move into the recipe with the logic.

[danceratopz]

Suggestion (anchored here because the rest of the loader move is a pure rename in the diff): let loaders/ and utils.py follow evm_tools all the way into the testing package, in this PR, rather than stopping at ethereum_spec_tools top level.

Justification: After the move they have zero consumers left in the spec package: every user of loaders/ and utils.py is in packages/testing (11 import sites) or tests/json_loader (one import, Load in helpers/load_blockchain_tests.py). The only spec-internal use is this file importing utils.parse_hex_or_int, and it moves with them. forks.py is different and should stay, it is real spec API.

Why now? The import path already breaks in this PR either way: on base these lived at ethereum_spec_tools.evm_tools.loaders and evm_tools/utils.py, so any external consumer has to update regardless. Landing them at execution_testing.evm_tools.loaders and evm_tools/utils.py now means one break instead of two.

---

What the full move buys, beyond the cleaner boundary:

• 13 of the 16 new vulture whitelist entries go away outright (Load, the seven ForkLoad.has_*, the five utils helpers leave src/); the remaining three are covered by extending the deadcode scan (other comment).
• The "ethereum_spec_tools.loaders" line added to the setuptools packages list can be dropped again.
• The new fork-selection branch in tests/json_loader/helpers/select_tests.py:79-84 can be reverted; it exists only because these files stayed behind.
• The PR's story simplifies from "move evm_tools, but split out its loaders and utils" to "move evm_tools wholesale".

Cost is the same mechanical class as the rest of the PR: five files moved plus roughly a dozen import-site rewrites.

[danceratopz]

The layout in the three states, for the visual argument.

1. Base (forks/amsterdam), everything under one roof:

src/ethereum_spec_tools/
├── __init__.py, forks.py, patch_tool.py, sync.py, py.typed
└── evm_tools/
    ├── __init__.py, __main__.py, daemon.py
    ├── utils.py                                    ◄ inside evm_tools
    ├── b11r/        (__init__.py, b11r_types.py)
    ├── loaders/     (__init__.py, fixture_loader.py,   ◄ inside evm_tools
    │                 fork_loader.py, transaction_loader.py)
    ├── statetest/   (__init__.py)
    └── t8n/         (__init__.py, block_environment.py, cli.py,
                      result.py, evm_trace/ ×5)

2. This PR as it stands, evm_tools moves but loaders/ and utils.py are split out and promoted to spec-tools top level:

src/ethereum_spec_tools/
├── __init__.py, forks.py, patch_tool.py, sync.py, py.typed
├── loaders/     (__init__.py, fixture_loader.py,   ◄ promoted, stayed behind
│                 fork_loader.py, transaction_loader.py)
└── utils.py                                        ◄ promoted, stayed behind

packages/testing/src/execution_testing/
└── evm_tools/
    ├── __init__.py, __main__.py, daemon.py
    ├── b11r/        (__init__.py, b11r_types.py)
    ├── statetest/   (__init__.py)
    ├── t8n/         (__init__.py, block_environment.py, cli.py,
    │                 result.py, evm_trace/ ×5)
    └── tests/       (×3, moved from tests/evm_tools/)

3. With the suggested move, evm_tools moves wholesale and keeps its base-internal structure at the new address:

src/ethereum_spec_tools/
├── __init__.py
├── forks.py                                        ◄ stays: real spec API
└── patch_tool.py, sync.py, py.typed

packages/testing/src/execution_testing/
└── evm_tools/
    ├── __init__.py, __main__.py, daemon.py
    ├── utils.py                                    ◄ back where it was
    ├── b11r/        (__init__.py, b11r_types.py)
    ├── loaders/     (__init__.py, fixture_loader.py,   ◄ back where it was
    │                 fork_loader.py, transaction_loader.py)
    ├── statetest/   (__init__.py)
    ├── t8n/         (__init__.py, block_environment.py, cli.py,
    │                 result.py, evm_trace/ ×5)
    └── tests/       (×3)

Why the picture matters: state 3 is not a new reorganization on top of the PR; it is the PR's own move applied uniformly. evm_tools in state 3 has the identical internal shape it had on base (utils.py and loaders/ in their old positions relative to t8n/, b11r/, statetest/); only the package prefix changed. State 2 invents a layout that never existed before, loaders/ and utils.py at spec-tools top level. The leftover src/ethereum_spec_tools/ in state 3 is pure spec-serving tooling: fork enumeration (forks.py), lint, new-fork scaffolding, docs, sync.