test(engine-tree): flaky test_state_root post-success SIGABRT (pthread lock)

[nekomoto911]

Summary

reth-engine-tree::tree::payload_processor::tests::test_state_root is flaky under nextest package-level / CI unit shards:

1. Test body completes successfully (assert_eq! / libtest prints ok).
2. Process then aborts with:

   pthread lock: Invalid argument
   (test aborted with signal 6: SIGABRT)
   

3. nextest retries (default profile: 2 retries / 3 attempts total) often eventually pass, so CI can go green or red nondeterministically.

This is unrelated to tx-filter / gas packing work (e.g. PR #414). It can still fail the required unit / test / ethereum (*/*) jobs and force cancel of sibling shards.

Evidence

CI (PR #414 head 726ee11)

• Run: https://github.com/Galxe/gravity-reth/actions/runs/30779756650
• Job: unit / test / ethereum (1/2)
  https://github.com/Galxe/gravity-reth/actions/runs/30779756650/job/91581944771
• Summary: 1200 tests run: 1199 passed, 1 failed
• Failure (3 nextest attempts):

  TRY N ABRT  reth-engine-tree tree::payload_processor::tests::test_state_root
  stdout:
    test tree::payload_processor::tests::test_state_root ... ok
  stderr:
    pthread lock: Invalid argument
    (test aborted with signal 6: SIGABRT)
  

Local repro (full package, nextest)

Isolated single-test runs usually pass. Full package:

cargo nextest run -p reth-engine-tree --lib

Reproduces ABRT on test_state_root under concurrent package load (same post-ok pattern). Separately, tree::tests::test_tree_persist_blocks often hits the nextest 60s slow-timeout under the same full-package stress (may be independent; see Notes).

Suspected root cause

Lifecycle of background work in the legacy PayloadProcessor path:

1. WorkloadExecutor uses a process-global OnceLock tokio runtime plus a per-instance rayon pool (crates/engine/tree/src/tree/payload_processor/executor.rs).
2. spawn starts multiproof / sparse-trie / proof / prewarm / tx-iterator via spawn_blocking but drops JoinHandles (fire-and-forget).
3. test_state_root builds a temporary MDBX factory, drives state updates, awaits handle.state_root(), asserts, then drops handle → processor → provider → factory.
4. Background tasks can still hold DB / trie resources while the test process tears down MDBX and static runtime → classic post-success pthread lock + SIGABRT.

So this is a teardown race, not a wrong state-root value (the body already matched serial root).

Upstream reth (paradigmxyz/reth / local ../reth)

Not the same bug on the same code path.

Upstream has moved state-root off the old PayloadProcessor multiproof stack into state_root_strategy + reth_tasks::Runtime::test() (owned runtime per test, cancel guards). The equivalent test is roughly state_root_task_matches_serial_root. There is no identical payload_processor::tests::test_state_root + static WorkloadExecutor OnceLock path.

So: do not assume a one-line cherry-pick from tip reth; gravity still carries the older engine-tree shape.

Suggested directions (short → long)

1. Short (targeted): Keep JoinHandles on PayloadHandle and join them on drop (after terminating prewarm / dropping multiproof sender). Local experiment after this change: full-package runs showed stable PASS for test_state_root (no ABRT); verify production drop paths still OK (block_on / block_in_place when already inside a runtime).
2. Short (CI hygiene): Optionally re-run failed unit jobs; nextest retries already paper over some flakes.
3. Medium: Align with upstream — Runtime::test() / proper cancel semantics for state-root tasks; avoid process-global OnceLock RT for tests.
4. Do not couple this to product PRs (e.g. gas last-gate) unless the PR intentionally touches engine-tree.

Acceptance

• cargo nextest run -p reth-engine-tree --lib green across ≥5 consecutive full-package runs (no ABRT on test_state_root).
• CI unit job no longer fails solely on this post-success SIGABRT.
• Optional: file/follow-up if test_tree_persist_blocks 60s timeout remains under full-package stress.

Notes

• nextest default profile: retries = { backoff = "exponential", count = 2, ... } and slow-timeout = { period = "30s", terminate-after = 2 } (effective ~60s kill) in .config/nextest.toml.
• Related noise under the same stress: tree::tests::test_tree_persist_blocks timing out — may be independent load/deadlock; track separately if it persists after the ABRT fix.

Refs

• PR that surfaced it: fix(tx-filter): apply block gas as last gate at Beta (discard non-fitting) #414
• Unit failure run: https://github.com/Galxe/gravity-reth/actions/runs/30779756650