feat(issue4): add real NCCL bitwise reproducibility diagnostics

[liu66-qing]

Summary

Adds a reproducible diagnostic workflow for the 2026 Issue 4 acceptance
criteria. The tool runs the same collective in independent torchrun
process groups, captures every rank/call as raw bytes, and localizes the
first run-to-run divergence.

Why this approach

A NumPy reduction-order simulation cannot establish NCCL reproducibility
because it does not exercise NCCL algorithm selection, topology discovery,
CUDA kernels, or communicator initialization. Repeating an operation inside
one communicator also cannot establish run-to-run behavior.

This implementation therefore:

• launches fresh NCCL communicators for every run;
• generates fixed but rank-distinct inputs;
• supports AllReduce and Reduce-Scatter;
• compares shape, dtype, raw bytes, SHA-256, changed bytes/bits, absolute
  error, and signed IEEE ULP distance;
• reports the first divergent call, run, rank, byte, and element;
• records PyTorch/CUDA/NCCL/GPU metadata and effective NCCL overrides;
• provides a controlled algorithm/protocol matrix without hiding unsupported
  combinations.

Files

• worker.py: one real distributed NCCL capture
• diagnose.py: independent-run orchestration and report generation
• sweep.py: unified NCCL_ALGO × NCCL_PROTO experiment matrix
• core.py: hardware-independent comparison/report engine
• test_core.py: CPU unit tests for equality, first-offset localization,
  positive/negative ULP distance, and first-call localization
• README.md: experiment protocol and three-dimensional analysis guidance

Correctness and safety

The documentation intentionally avoids undocumented controls such as
NCCL_DETERMINISTIC and does not label Ring/Tree/PAT/NVLS deterministic by
assumption. NVIDIA documents NCCL_ALGO and NCCL_PROTO as selection/debug
controls and warns that forcing unsupported LL128 may corrupt data, so
unsupported configurations are surfaced as failures.

Raw captures are deleted after report creation by default to bound retained
storage. The README documents capture scaling and requires changing one
experimental factor at a time.

Validation

Completed locally and on a real four-GPU NCCL host.

python -m unittest -v test_core.py
Ran 4 tests in 0.005s — OK

python -m py_compile core.py diagnose.py worker.py sweep.py test_core.py
git diff --check

Measured environment:

• 4 × RTX 3090, dual-NUMA PCIe topology (PXB within pairs, SYS across pairs)
• PyTorch 2.8.0+cu128, CUDA 12.8, NCCL 2.27.3
• 5 independent launches per configuration
• 1 KiB, 64 KiB, 1 MiB, and 16 MiB AllReduce messages
• AllReduce: automatic/Ring/Tree × automatic/Simple/LL
• Reduce-Scatter: automatic/Ring × automatic/Simple/LL

All valid same-configuration runs were bitwise identical. Tree Reduce-Scatter
was rejected with ncclInvalidUsage and was not silently replaced. A
controlled Ring-to-Tree change produced a first difference at call 0, rank 0,
float32 element 6, with maximum absolute error 9.5367431640625e-07.

Full methodology, bounded conclusions, topology, hashes, and machine-readable
evidence are in EXPERIMENT.md and results/autodl-4x3090.json.

Closes #4