fix(consume): harden enginex hive reporting against connection errors - #3335
Draft
danceratopz wants to merge 2 commits into
Draft
fix(consume): harden enginex hive reporting against connection errors#3335danceratopz wants to merge 2 commits into
danceratopz wants to merge 2 commits into
Conversation
Complementary hardening for the TIME_WAIT ephemeral-port exhaustion that caused sporadic Nethermind consume-enginex failures and silent test loss (the root fix pools hive API connections in ethereum-hive): - Retry hive API calls in the per-test reporting paths (start_test and end_test in the hive_test fixture, register_multi_test_client in the enginex conftest) on connection errors with exponential backoff, so seconds-long EADDRNOTAVAIL bursts no longer error tests or lose results. - Detect silent test loss: each xdist worker counts the test results it successfully reports to hive; the last worker to finish aggregates the counts and compares them against the number of collected tests before ending the suite. On a shortfall it reports a failing 'reported-test-count-check' meta-test case to hive (making the loss visible in the hive UI, where lost tests are otherwise silently missing) and errors the pytest session. Enabled for enginex via config.assert_reported_test_count.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #3335 +/- ##
================================================
Coverage 93.50% 93.50%
================================================
Files 624 624
Lines 37070 37070
Branches 3394 3394
================================================
Hits 34661 34661
Misses 1653 1653
Partials 756 756
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Without an __init__.py in plugins/pytest_hive/, pytest's prepend import mode resolved the new pytest_hive/tests/ package as top-level 'tests' — colliding with plugins/shared/tests/ (same bare-parent layout) and breaking collection of test_address_stubs.py with ModuleNotFoundError. With the package regular, its tests import fully qualified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP: please don't review yet; pending self-review
Related: ethereum/hive-python-api#18
Description
Complementary simulator-side hardening for the TIME_WAIT ephemeral-port exhaustion behind the sporadic Nethermind
consume-enginexdashboard failures on 2026-07-18/19 (4–5 spurious "Test was terminated by host" fails and 85/40 silently missing test results per run). The root fix pools the Hive API connections inethereum-hive(hive-python-api#18); this PR makes the enginex reporting paths resilient and makes any remaining silent test loss loud:start_test/end_testin thehive_testfixture andregister_multi_test_clientin the enginex conftest (5 attempts, 0.5 s initial backoff, ~7.5 s total). Connection errors are raised before the request is sent, so retrying cannot duplicate a call. Works against the releasedethereum-hive==0.1.0a5, i.e. needs no package release; with the package fix it is belt-and-suspenders.config.assert_reported_test_count): each xdist worker counts the test results it successfully reports to hive (plus setup-skipped tests); the last worker to finish aggregates all workers' counts and compares againstsession.testscollectedbefore ending the suite. On a shortfall it (a) reports a failingreported-test-count-checkmeta-test case to hive, making the loss visible in the hive UI where lost tests are otherwise silently missing, and (b) raisesHiveReportedTestCountError, erroring the pytest session. Skipped on interrupted runs (-x/--maxfail/^C); only a shortfall fires, so skip edge cases cannot false-fail. This also surfaces the known singleton-group client-startup flake instead of letting it silently shrink the listed test count.Helpers live in
plugins/pytest_hive/reporting.py(a non-plugin module so the enginex conftest can import it without disabling assert rewriting for the plugin), with unit tests covering the loss path a healthy run cannot exercise.Validated with hive
--dev+ nethermind (12,888 Cancun tests,-n 4): verdicts identical to baseline, check reports 12,888/12,888, peak TIME_WAIT toward the hive API 9,684 → 66 with the patched package.Follow-up (separate PR once
ethereum-hive==0.1.0a6is on PyPI): bump the pin inpackages/testing/pyproject.tomlto>=0.1.0a6,<1.0.0.Related Issues or PRs
Root fix: hive-python-api#18 (root-cause diagnosis: nethermind enginex fails 2026-07-18/19 — simulator ephemeral-port exhaustion, not a client bug).
Checklist
just static<type>(<area>): <title>, where<type>and<area>come from an appropriateC-<type>, respectivelyA-<area>, label. The title should match the target squash commit message.Cute Animal Picture