Summary
The ARM64 linux64_tsan build intermittently fails in make check while running bls_verify_contribution_shares_honest_tests. ThreadSanitizer reports two BLS worker threads racing inside mimalloc initialization/allocation metadata.
Failing CI job: https://github.com/dashpay/dash/actions/runs/30781028052/job/91585818041
Failure
The source build completes, then test/bls_tests.cpp.test fails:
test/bls_tests.cpp(641): Entering test case "bls_verify_contribution_shares_honest_tests"
/usr/bin/llvm-symbolizer-19: error: '[anon:mimalloc]': No such file or directory
==================
WARNING: ThreadSanitizer: data race (pid=51626)
Write of size 8 at 0xff5f7f024400 by thread T8:
#0 memcpy
#1 _mi_heap_init
#2 CBLSWorker::GenerateContributions(...) src/bls/bls_worker.cpp:103:37
Previous write of size 8 at 0xff5f7f024400 by thread T5:
#0 memset
#1 _mi_meta_zalloc
#2 CBLSWorker::GenerateContributions(...) src/bls/bls_worker.cpp:103:37
Location is global '??' at 0xff5f7f021000 ([anon:mimalloc]+0x3400)
Thread T8 'd-bls-work-2' ...
Thread T5 'd-bls-work-0' ...
SUMMARY: ThreadSanitizer: data race ... in memcpy
make[3]: *** [Makefile:27917: test/bls_tests.cpp.test] Error 1
make[2]: *** [Makefile:25326: check-am] Error 2
make[1]: *** [Makefile:24991: check-recursive] Error 1
Both threads were created by CBLSWorker::Start() for FuncVerifyContributionSharesHonest(). They concurrently reach SecretKeyShare() at src/bls/bls_worker.cpp:103, with the reported conflicting writes in _mi_heap_init and _mi_meta_zalloc in the anonymous mimalloc mapping.
Why this appears intermittent/pre-existing
This was observed on PR #7400 at head 7be785489814cad1aa4dde0347b49d68a3d7fc65, but that PR changes wallet/coin-selection, RPC, Qt, and related tests only. It does not change BLS, mimalloc, the depends configuration, or the TSAN CI setup.
The contemporaneous develop run completed both linux64_tsan-build and linux64_tsan-test successfully: https://github.com/dashpay/dash/actions/runs/30775250510 (head f1dde51ed0d0f1de13678159ca7edd3d814bfe23). The conflicting worker-thread initialization path is timing-dependent, which is consistent with an intermittent allocator-initialization race rather than a regression from #7400.
Environment / reproduction
CI environment from the failing job:
- Workflow target:
linux64_tsan
- Runner:
ubuntu-24.04-arm / AArch64
- Container:
ghcr.io/dashpay/dash/dashcore-ci-runner:develop
- Compiler:
clang-19 / clang++-19
- Configure sanitizer option:
--with-sanitizers=thread
- Depends:
HOST=aarch64-linux-gnu MULTIPROCESS=1 CC=clang-19 CXX=clang++-19
Run the same CI target, potentially repeatedly because the failure is timing-sensitive:
make -j"$(nproc)" -C depends HOST=aarch64-linux-gnu MULTIPROCESS=1 CC=clang-19 CXX=clang++-19
BUILD_TARGET=linux64_tsan
source ./ci/dash/matrix.sh
./ci/dash/build_src.sh
After a TSAN build, the relevant unit-test suite can be stressed directly with:
build-ci/dashcore-linux64_tsan/src/test/test_dash \
--run_test=bls_tests/bls_verify_contribution_shares_honest_tests
Possible investigation directions
- Determine whether mimalloc requires eager single-threaded process initialization before the BLS worker pool starts allocating.
- Check whether the bundled mimalloc version/configuration has a known ARM64 TSAN initialization issue or needs a version update/fix.
- Improve symbolization for
[anon:mimalloc] so future reports resolve the allocator source lines.
Summary
The ARM64
linux64_tsanbuild intermittently fails inmake checkwhile runningbls_verify_contribution_shares_honest_tests. ThreadSanitizer reports two BLS worker threads racing inside mimalloc initialization/allocation metadata.Failing CI job: https://github.com/dashpay/dash/actions/runs/30781028052/job/91585818041
Failure
The source build completes, then
test/bls_tests.cpp.testfails:Both threads were created by
CBLSWorker::Start()forFuncVerifyContributionSharesHonest(). They concurrently reachSecretKeyShare()atsrc/bls/bls_worker.cpp:103, with the reported conflicting writes in_mi_heap_initand_mi_meta_zallocin the anonymous mimalloc mapping.Why this appears intermittent/pre-existing
This was observed on PR #7400 at head
7be785489814cad1aa4dde0347b49d68a3d7fc65, but that PR changes wallet/coin-selection, RPC, Qt, and related tests only. It does not change BLS, mimalloc, the depends configuration, or the TSAN CI setup.The contemporaneous
developrun completed bothlinux64_tsan-buildandlinux64_tsan-testsuccessfully: https://github.com/dashpay/dash/actions/runs/30775250510 (headf1dde51ed0d0f1de13678159ca7edd3d814bfe23). The conflicting worker-thread initialization path is timing-dependent, which is consistent with an intermittent allocator-initialization race rather than a regression from #7400.Environment / reproduction
CI environment from the failing job:
linux64_tsanubuntu-24.04-arm/ AArch64ghcr.io/dashpay/dash/dashcore-ci-runner:developclang-19/clang++-19--with-sanitizers=threadHOST=aarch64-linux-gnu MULTIPROCESS=1 CC=clang-19 CXX=clang++-19Run the same CI target, potentially repeatedly because the failure is timing-sensitive:
After a TSAN build, the relevant unit-test suite can be stressed directly with:
Possible investigation directions
[anon:mimalloc]so future reports resolve the allocator source lines.