backport: bitcoin#18288, #21852, #21864, #21865, #23008, #23619, #24384, #24522, #24705, #26683, #26843, #27436, #27495, #27699, #27737, #28185, #29676, #29742, #29800, #29837, #31592 #7642
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Copyright (c) 2020-present The Bitcoin Core developers | ||
| # Distributed under the MIT software license, see the accompanying | ||
| # file COPYING or http://www.opensource.org/licenses/mit-license.php. | ||
|
|
||
| export LC_ALL=C.UTF-8 | ||
|
|
||
| export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04" | ||
| LIBCXX_DIR="/msan/cxx_build/" | ||
| export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls" | ||
| LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument" | ||
|
Comment on lines
+10
to
+12
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- repository conventions ---'
find /tmp/coderabbit-repo-knowledge/dashpay-dash-9d8ac790 -maxdepth 2 -type f -name '*.md' -print
printf '%s\n' '--- target script ---'
cat -n ci/test/00_setup_env_native_fuzz_with_msan.sh
printf '%s\n' '--- directly referenced MSan setup symbols ---'
rg -n -C 4 'BASE_SCRATCH_DIR|msan/build|cxx_build|LIBCXX_DIR|MSAN_FLAGS|LIBCXX_FLAGS' ciRepository: dashpay/dash Length of output: 11646 🏁 Script executed: printf '%s\n' '--- applicable conventions ---'
cat /tmp/coderabbit-repo-knowledge/dashpay-dash-9d8ac790/conventions/repo-wide.md
printf '%s\n' '--- install flow around MSan bootstrap ---'
cat -n ci/test/04_install.sh | sed -n '1,50p;115,150p'
printf '%s\n' '--- CI command and path bindings ---'
rg -n -C 5 'DOCKER_EXEC|DOCKER_EXEC_ROOT|BASE_SCRATCH_DIR|ci/test/00_setup_env|setup_env' ci/test ci | head -240
printf '%s\n' '--- MSan container/build references ---'
rg -n -C 4 'cxx_build|ci-msan|llvm-project|MemoryWithOrigins|/msan|/cxx_build' .github ci contribRepository: dashpay/dash Length of output: 25449 🏁 Script executed: printf '%s\n' '--- target selection and execution order ---'
cat -n ci/test_run_all.sh
cat -n ci/test/00_setup_env.sh | sed -n '1,115p'
printf '%s\n' '--- install execution and mounts ---'
cat -n ci/test/04_install.sh | sed -n '45,115p'
printf '%s\n' '--- exact references to the target and libc++ paths ---'
rg -n -C 6 '00_setup_env_native_fuzz_with_msan|ci_native_fuzz_msan|msan/cxx_build|cxx_build|ci-msan\.Dockerfile' . --glob '!*.md'Repository: dashpay/dash Length of output: 14316 🏁 Script executed: printf '%s\n' '--- command definitions and image selection ---'
rg -n -C 8 '(^|[[:space:]])(function[[:space:]]+)?(DOCKER_EXEC|CI_EXEC)[[:space:](=]|$)|CI_IMAGE_NAME_TAG|DOCKER_NAME_TAG' ci .github contrib
printf '%s\n' '--- build stage consuming the sanitizer flags ---'
rg -n -C 6 'DEP_OPTS|CXXFLAGS|MSAN_AND_LIBCXX_FLAGS|make .*configure|./configure|BASE_OUTDIR|msan' ci/test/05_before_script.sh ci/test/06_script.sh ci/test
printf '%s\n' '--- tracked files under CI container configuration ---'
git ls-files 'contrib/containers/ci/*' 'ci/test/*msan*' 'ci/test/*fuzz*'Repository: dashpay/dash Length of output: 22713 🏁 Script executed: printf '%s\n' '--- every DOCKER_EXEC definition or call ---'
rg -n -C 3 'DOCKER_EXEC' .
printf '%s\n' '--- every image variable reference ---'
rg -n -C 3 'CI_IMAGE_NAME_TAG|DOCKER_NAME_TAG' ci .github contrib
printf '%s\n' '--- remaining build stages ---'
cat -n ci/test/05_before_script.sh | sed -n '1,80p'Repository: dashpay/dash Length of output: 6131 Use the libc++ build output in The MSan bootstrap commands use 🤖 Prompt for AI AgentsThere was a problem hiding this comment. The missing libc++ output is valid, but There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- fuzz MSan setup ---'
cat -n ci/test/00_setup_env_native_fuzz_with_msan.sh
printf '%s\n' '--- MSan image definition ---'
cat -n contrib/containers/ci/ci-msan.Dockerfile
printf '%s\n' '--- MSan target references ---'
rg -n -C 4 'native_fuzz_with_msan|native_fuzz_msan|linux64_msan|ci-msan|CI_IMAGE_NAME_TAG|LIBCXX_DIR' \
ci .github contrib
printf '%s\n' '--- image build/selection conventions ---'
rg -n -C 3 'ci-[A-Za-z0-9_-]+\.Dockerfile|CI_IMAGE_NAME_TAG=.*ci-|dockerfile-x' \
ci contrib .githubLength of output: 13049
The fuzz setup uses The finding remains valid, but its proposed 🐇🔍 ✏️ Learnings added
You are interacting with an AI system.
Comment on lines
+9
to
+12
There was a problem hiding this comment. 🔴 Blocking: Missing libc++ producer for the backported fuzz-MSan configuration This configuration points at source: ['codex']
Collaborator
Author
There was a problem hiding this comment. that's correct finding and it's fixed mainstream in bitcoin#33099 |
||
| export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}" | ||
|
|
||
| export CONTAINER_NAME="ci_native_fuzz_msan" | ||
| export PACKAGES="ninja-build" | ||
| # BDB generates false-positives and will be removed in future | ||
| export DEP_OPTS="DEBUG=1 NO_BDB=1 NO_QT=1 CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'" | ||
| export GOAL="install" | ||
| # _FORTIFY_SOURCE is not compatible with MSAN. | ||
| export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,memory CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE -U_FORTIFY_SOURCE'" | ||
| export USE_MEMORY_SANITIZER="true" | ||
| export RUN_UNIT_TESTS="false" | ||
| export RUN_FUNCTIONAL_TESTS="false" | ||
| export RUN_FUZZ_TESTS=true | ||
| export CCACHE_MAXSIZE=250M | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Copyright (c) 2020-present The Bitcoin Core developers | ||
| # Distributed under the MIT software license, see the accompanying | ||
| # file COPYING or http://www.opensource.org/licenses/mit-license.php. | ||
|
|
||
| export LC_ALL=C.UTF-8 | ||
|
|
||
| export CONTAINER_NAME="ci_native_msan" | ||
| export HOST=x86_64-pc-linux-gnu | ||
| # Built into the CI image by contrib/containers/ci/ci-msan.Dockerfile so that it | ||
| # lands in a cached layer instead of being rebuilt on every run. Upstream does | ||
| # the equivalent in ci/test/01_base_install.sh, which Dash does not have. | ||
| LIBCXX_DIR="/cxx_build/" | ||
| export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls" | ||
| LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument" | ||
| export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}" | ||
|
|
||
| # BDB generates false-positives and will be removed in future | ||
| export DEP_OPTS="DEBUG=1 NO_BDB=1 NO_QT=1 NO_UPNP=1 NO_NATPMP=1 CC=clang-19 CXX=clang++-19 CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'" | ||
| export GOAL="install" | ||
| # CC/CXX/CFLAGS/CXXFLAGS are repeated here rather than left to depends' | ||
| # config.site. Upstream dropped them in bitcoin#29800 because its | ||
| # ci/test/03_test_script.sh sets CONFIG_SITE explicitly (bitcoin#26683); | ||
| # ci/dash/build_src.sh still uses the older --prefix form, and this is the only | ||
| # target whose compiler differs from the system default, so nothing else in | ||
| # Dash's CI exercises that hand-off. | ||
| # _FORTIFY_SOURCE is not compatible with MSAN. | ||
| # --with-asm=no and --with-backend=easy keep secp256k1 and relic off | ||
| # hand-written assembly, which MSan cannot see through. | ||
| # --disable-mimalloc selects the plain malloc secure allocator; mimalloc seeds | ||
| # itself with a raw getrandom syscall before main(), which MSan cannot track. | ||
| export BITCOIN_CONFIG="--with-sanitizers=memory --with-gui=no --without-bdb --with-sqlite \ | ||
| --with-asm=no --with-backend=easy --disable-mimalloc \ | ||
|
There was a problem hiding this comment.
When AGENTS.md reference: AGENTS.md:L186-L191 Useful? React with 👍 / 👎.
Comment on lines
+29
to
+34
There was a problem hiding this comment. 🟡 Suggestion: Missing prerequisite: bitcoin#29803 for bitcoin#29742 Upstream merge b5d2118 for bitcoin#29803 is an ancestor of bitcoin#29742 and added source: ['codex'] |
||
| CC=clang-19 CXX=clang++-19 CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}' \ | ||
|
Comment on lines
+31
to
+35
There was a problem hiding this comment. 🔴 Blocking: Implement the mimalloc switch instead of passing an ignored option
source: ['codex']
Collaborator
Author
There was a problem hiding this comment. These fixes exists but out-of-scope of this PR See commit There was a problem hiding this comment. Resolved in this update — Implement the mimalloc switch instead of passing an ignored option no longer present. Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread. |
||
| CPPFLAGS='-U_FORTIFY_SOURCE'" | ||
| export USE_MEMORY_SANITIZER="true" | ||
| export TEST_RUNNER_TIMEOUT_FACTOR=40 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # syntax = devthefuture/dockerfile-x | ||
|
|
||
| FROM ./ci.Dockerfile | ||
|
|
||
| # MemorySanitizer reports any read of memory it did not watch being written, | ||
| # so the C++ standard library has to be instrumented too. A stock libstdc++ or | ||
| # libc++ produces a flood of reports that cannot be told apart from real ones. | ||
| # | ||
| # Upstream builds this in ci/test/01_base_install.sh, which runs inside its | ||
| # docker build. Dash has no such script, so it lives here instead, which keeps | ||
| # the result in a cached image layer rather than rebuilding it on every run. | ||
|
|
||
| USER root | ||
|
|
||
| ARG LLVM_VERSION=19 | ||
| ARG LLVM_TAG=llvmorg-19.1.7 | ||
|
|
||
| RUN set -ex; \ | ||
| apt-get update && apt-get install ${APT_ARGS} ninja-build; \ | ||
| rm -rf /var/lib/apt/lists/*; \ | ||
| git clone --depth=1 -b "${LLVM_TAG}" https://github.com/llvm/llvm-project /llvm-project; \ | ||
| cmake -G Ninja -B /cxx_build/ \ | ||
| -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DLLVM_USE_SANITIZER=MemoryWithOrigins \ | ||
| -DCMAKE_C_COMPILER=clang-${LLVM_VERSION} \ | ||
| -DCMAKE_CXX_COMPILER=clang++-${LLVM_VERSION} \ | ||
| -DLLVM_TARGETS_TO_BUILD=Native \ | ||
| -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF \ | ||
| -DLIBCXXABI_USE_LLVM_UNWINDER=OFF \ | ||
| -DLIBCXX_HARDENING_MODE=debug \ | ||
| -S /llvm-project/runtimes; \ | ||
| ninja -C /cxx_build/; \ | ||
| du -sh /llvm-project; \ | ||
| rm -rf /llvm-project; | ||
|
|
||
| USER dash |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the fuzz-with-MSan configuration is enabled, its compiler and linker flags reference
/msan/cxx_build, but the only instrumented libc++ added by this series is created at/cxx_buildincontrib/containers/ci/ci-msan.Dockerfile; a repo-wide search finds nothing that creates the former path. Sourcing this config in the dedicated MSan image will therefore fail as soon as Clang looks for the libc++ headers or libraries, so this configuration needs the same Dash path adaptation as00_setup_env_native_msan.sh.AGENTS.md reference: AGENTS.md:L186-L191
Useful? React with 👍 / 👎.