Skip to content

Intermittent ARM64 TSAN race in mimalloc during BLS contribution test #7530

Description

@thepastaclaw

Summary

The ARM64 linux64_tsan build intermittently fails in make check while running bls_verify_contribution_shares_honest_tests. ThreadSanitizer reports two BLS worker threads racing inside mimalloc initialization/allocation metadata.

Failing CI job: https://github.com/dashpay/dash/actions/runs/30781028052/job/91585818041

Failure

The source build completes, then test/bls_tests.cpp.test fails:

test/bls_tests.cpp(641): Entering test case "bls_verify_contribution_shares_honest_tests"
/usr/bin/llvm-symbolizer-19: error: '[anon:mimalloc]': No such file or directory
==================
WARNING: ThreadSanitizer: data race (pid=51626)
  Write of size 8 at 0xff5f7f024400 by thread T8:
    #0 memcpy
    #1 _mi_heap_init
    #2 CBLSWorker::GenerateContributions(...) src/bls/bls_worker.cpp:103:37

  Previous write of size 8 at 0xff5f7f024400 by thread T5:
    #0 memset
    #1 _mi_meta_zalloc
    #2 CBLSWorker::GenerateContributions(...) src/bls/bls_worker.cpp:103:37

  Location is global '??' at 0xff5f7f021000 ([anon:mimalloc]+0x3400)

  Thread T8 'd-bls-work-2' ...
  Thread T5 'd-bls-work-0' ...

SUMMARY: ThreadSanitizer: data race ... in memcpy
make[3]: *** [Makefile:27917: test/bls_tests.cpp.test] Error 1
make[2]: *** [Makefile:25326: check-am] Error 2
make[1]: *** [Makefile:24991: check-recursive] Error 1

Both threads were created by CBLSWorker::Start() for FuncVerifyContributionSharesHonest(). They concurrently reach SecretKeyShare() at src/bls/bls_worker.cpp:103, with the reported conflicting writes in _mi_heap_init and _mi_meta_zalloc in the anonymous mimalloc mapping.

Why this appears intermittent/pre-existing

This was observed on PR #7400 at head 7be785489814cad1aa4dde0347b49d68a3d7fc65, but that PR changes wallet/coin-selection, RPC, Qt, and related tests only. It does not change BLS, mimalloc, the depends configuration, or the TSAN CI setup.

The contemporaneous develop run completed both linux64_tsan-build and linux64_tsan-test successfully: https://github.com/dashpay/dash/actions/runs/30775250510 (head f1dde51ed0d0f1de13678159ca7edd3d814bfe23). The conflicting worker-thread initialization path is timing-dependent, which is consistent with an intermittent allocator-initialization race rather than a regression from #7400.

Environment / reproduction

CI environment from the failing job:

  • Workflow target: linux64_tsan
  • Runner: ubuntu-24.04-arm / AArch64
  • Container: ghcr.io/dashpay/dash/dashcore-ci-runner:develop
  • Compiler: clang-19 / clang++-19
  • Configure sanitizer option: --with-sanitizers=thread
  • Depends: HOST=aarch64-linux-gnu MULTIPROCESS=1 CC=clang-19 CXX=clang++-19

Run the same CI target, potentially repeatedly because the failure is timing-sensitive:

make -j"$(nproc)" -C depends HOST=aarch64-linux-gnu MULTIPROCESS=1 CC=clang-19 CXX=clang++-19
BUILD_TARGET=linux64_tsan
source ./ci/dash/matrix.sh
./ci/dash/build_src.sh

After a TSAN build, the relevant unit-test suite can be stressed directly with:

build-ci/dashcore-linux64_tsan/src/test/test_dash \
  --run_test=bls_tests/bls_verify_contribution_shares_honest_tests

Possible investigation directions

  • Determine whether mimalloc requires eager single-threaded process initialization before the BLS worker pool starts allocating.
  • Check whether the bundled mimalloc version/configuration has a known ARM64 TSAN initialization issue or needs a version update/fix.
  • Improve symbolization for [anon:mimalloc] so future reports resolve the allocator source lines.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions