fix: Clamp the depth used to index selectBranch's key byte

[bthomee]

Part 4/8 of a stack. Base: part 3 (bthomee/shamap-stack-03-reject-inner-node-leaf-depth).

selectBranch reads the key byte at depth / 2, which is out of bounds for a
32-byte key once the depth reaches 64. Every branch selection in the map
funnels through here, so this is the one place a stray depth can turn into a
bad read. Assert the precondition for callers, then clamp anyway: a wrong
answer for an input that should never occur is better than reading past the
buffer. Verified under ASan that the unclamped form reads one byte past a
32-byte allocation while the clamped form does not.

deserializeSHAMapNodeID gets the same mask check isPrefixOf already
performs, pulled into a shared isPrefixOfAtDepth helper.

Tests cover the depth-sensitivity of isPrefixOf, the guards that must hold
with asserts stripped, that deserializeSHAMapNodeID rejects an out-of-range
depth, and the clamp itself under both build configurations (EXPECT_DEATH
in a forked process when the assert is live, and the clamped result compared
against depth 63 when it is not). The clamp test also gates on
ENABLE_VOIDSTAR, not just NDEBUG: under Antithesis instrumentation
XRPL_ASSERT routes to a handler that records the hit but never aborts, so a
Debug build with voidstar enabled has NDEBUG undefined yet still hits the
same non-fatal assert as a release build, and without this gate would send
that configuration into the EXPECT_DEATH arm, where the forked child never
dies and the test fails, breaking the CI job that runs this suite under
-Dvoidstar=ON.

[Copilot]

Pull request overview

This PR hardens SHAMap node branch selection and node-ID deserialization against out-of-range depths, preventing potential out-of-bounds reads on 32-byte keys and centralizing prefix/mask validation logic.

Changes:

• Add an assertion + runtime clamp in selectBranch to prevent indexing past the end of the key when depth reaches (or exceeds) leaf depth.
• Factor prefix/mask checking into a shared isPrefixOfAtDepth helper and reuse it in isPrefixOf and deserializeSHAMapNodeID.
• Add focused unit tests covering prefix depth-sensitivity, deserialization depth rejection, leaf-depth child guards, and the selectBranch clamp behavior under different build configurations.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/libxrpl/shamap/SHAMapNodeID.cpp	Introduces isPrefixOfAtDepth, tightens deserialization validation, and clamps selectBranch depth to avoid OOB reads.
src/tests/libxrpl/shamap/SHAMapNodeID.cpp	Adds new test coverage for SHAMapNodeID prefix behavior, deserialization validation, and selectBranch depth clamping/assert behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.