Investigate Release vs Debug discrepancy in simg4ox A/B record diff list

[plexoos]

Summary

While fixing Release/PRODUCTION builds to honor configured record.npy output, we observed that compare_ab.py reports different A/B mismatch index lists between Debug and Release builds.

This is unexpected. The comparison already uses np.allclose(..., rtol=0.0, atol=1e-5), so ordinary optimization-level floating-point noise should not change the mismatch set. The observed differences include large record discrepancies, not small numeric drift.

Observed Behavior

For Geant4 11.3.2, the current expected Debug/non-production mismatch list is:

[14, 22, 32, 34, 40, 81, 85]

In a Release build where PRODUCTION is defined, after enabling configured record output, the observed mismatch list was:

[14, 16, 22, 32, 40, 67, 81]

Notable differences:

• 16 and 67 appear only in Release.
• 34 and 85 disappear in Release.
• Differences are not just near-threshold floating point noise; some inspected entries had large differences such as 420 vs 0 or sign/direction flips.

Why This Matters

compare_ab.py should ideally key expected differences on physics/runtime dependencies such as Geant4 version, not on CMake build type.

If PRODUCTION changes the photon histories, then some code currently compiled out as “debug” may have hidden side effects or may affect state evolution. That would make Release and Debug builds behaviorally different in a way we do not currently understand.

Reproduction

Configure and build Release:

cmake -S simphony/ -B build \
  -DCMAKE_INSTALL_PREFIX=/opt/simphony \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --target simg4ox -j$(nproc)

Run the integration test:

rm -rf build/tests/simg4ox/*
ctest --test-dir build -R '^Integration\.simg4ox$' --output-on-failure

Then inspect the printed ACTUAL_DIFF from compare_ab.py.

Expected Behavior

Debug and Release builds should produce the same A/B mismatch index list for the same:

• source revision
• config
• geometry
• macro
• Geant4 version
• CUDA/OptiX/runtime environment

Small numeric differences within the existing tolerance are acceptable, but the mismatch set should not change due only to PRODUCTION.

Suggested Investigation

• Compare Debug vs Release record.npy arrays directly for indices:
  • common mismatches: 14, 22, 32, 40, 81
  • Debug-only mismatches: 34, 85
  • Release-only mismatches: 16, 67
• Check whether any code under #ifndef PRODUCTION mutates simulation state rather than only recording/debugging.
• Audit sctx::point, sctx::trace, and sctx::end for side effects that may influence later propagation.
• Check whether sequence/tag/debug structures affect random consumption, flags, boundary state, or final photon state.
• Confirm whether Debug vs Release changes remain after keeping compare_ab.py expectations build-type independent.

Notes

During the Release record-output fix, record.npy production was made runtime-configured:

• DebugLite / explicit record: allocate and save records.
• Minimal / normal production: no record allocation and no record output.

[plexoos]

See #387