Skip to content

fix(consume): harden enginex hive reporting against connection errors - #3335

Draft
danceratopz wants to merge 2 commits into
ethereum:forks/amsterdamfrom
danceratopz:enginex-port-exhaustion
Draft

fix(consume): harden enginex hive reporting against connection errors#3335
danceratopz wants to merge 2 commits into
ethereum:forks/amsterdamfrom
danceratopz:enginex-port-exhaustion

Conversation

@danceratopz

@danceratopz danceratopz commented Aug 7, 2026

Copy link
Copy Markdown
Member

WIP: please don't review yet; pending self-review

Related: ethereum/hive-python-api#18

Description

Complementary simulator-side hardening for the TIME_WAIT ephemeral-port exhaustion behind the sporadic Nethermind consume-enginex dashboard failures on 2026-07-18/19 (4–5 spurious "Test was terminated by host" fails and 85/40 silently missing test results per run). The root fix pools the Hive API connections in ethereum-hive (hive-python-api#18); this PR makes the enginex reporting paths resilient and makes any remaining silent test loss loud:

  • Retry with backoff on connection errors in the per-test reporting paths: start_test/end_test in the hive_test fixture and register_multi_test_client in the enginex conftest (5 attempts, 0.5 s initial backoff, ~7.5 s total). Connection errors are raised before the request is sent, so retrying cannot duplicate a call. Works against the released ethereum-hive==0.1.0a5, i.e. needs no package release; with the package fix it is belt-and-suspenders.
  • Collected-vs-reported assertion (enginex only, via config.assert_reported_test_count): each xdist worker counts the test results it successfully reports to hive (plus setup-skipped tests); the last worker to finish aggregates all workers' counts and compares against session.testscollected before ending the suite. On a shortfall it (a) reports a failing reported-test-count-check meta-test case to hive, making the loss visible in the hive UI where lost tests are otherwise silently missing, and (b) raises HiveReportedTestCountError, erroring the pytest session. Skipped on interrupted runs (-x/--maxfail/^C); only a shortfall fires, so skip edge cases cannot false-fail. This also surfaces the known singleton-group client-startup flake instead of letting it silently shrink the listed test count.

Helpers live in plugins/pytest_hive/reporting.py (a non-plugin module so the enginex conftest can import it without disabling assert rewriting for the plugin), with unit tests covering the loss path a healthy run cannot exercise.

Validated with hive --dev + nethermind (12,888 Cancun tests, -n 4): verdicts identical to baseline, check reports 12,888/12,888, peak TIME_WAIT toward the hive API 9,684 → 66 with the patched package.

Follow-up (separate PR once ethereum-hive==0.1.0a6 is on PyPI): bump the pin in packages/testing/pyproject.toml to >=0.1.0a6,<1.0.0.

Related Issues or PRs

Root fix: hive-python-api#18 (root-cause diagnosis: nethermind enginex fails 2026-07-18/19 — simulator ephemeral-port exhaustion, not a client bug).

Checklist

  • Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
  • PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

429 Too Many Requests

Complementary hardening for the TIME_WAIT ephemeral-port exhaustion
that caused sporadic Nethermind consume-enginex failures and silent
test loss (the root fix pools hive API connections in ethereum-hive):

- Retry hive API calls in the per-test reporting paths (start_test and
  end_test in the hive_test fixture, register_multi_test_client in the
  enginex conftest) on connection errors with exponential backoff, so
  seconds-long EADDRNOTAVAIL bursts no longer error tests or lose
  results.
- Detect silent test loss: each xdist worker counts the test results it
  successfully reports to hive; the last worker to finish aggregates
  the counts and compares them against the number of collected tests
  before ending the suite. On a shortfall it reports a failing
  'reported-test-count-check' meta-test case to hive (making the loss
  visible in the hive UI, where lost tests are otherwise silently
  missing) and errors the pytest session. Enabled for enginex via
  config.assert_reported_test_count.
@danceratopz danceratopz added C-bug Category: this is a bug, deviation, or other problem A-test-consume Area: execution_testing.cli.pytest_commands.plugins.consume labels Aug 7, 2026
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.50%. Comparing base (f1aa970) to head (f8f5593).
⚠️ Report is 8 commits behind head on forks/amsterdam.

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #3335   +/-   ##
================================================
  Coverage            93.50%   93.50%           
================================================
  Files                  624      624           
  Lines                37070    37070           
  Branches              3394     3394           
================================================
  Hits                 34661    34661           
  Misses                1653     1653           
  Partials               756      756           
Flag Coverage Δ
unittests 93.50% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Without an __init__.py in plugins/pytest_hive/, pytest's prepend
import mode resolved the new pytest_hive/tests/ package as top-level
'tests' — colliding with plugins/shared/tests/ (same bare-parent
layout) and breaking collection of test_address_stubs.py with
ModuleNotFoundError. With the package regular, its tests import fully
qualified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-consume Area: execution_testing.cli.pytest_commands.plugins.consume C-bug Category: this is a bug, deviation, or other problem

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant