feat(issue4): bitwise determinism diagnostic tool for NCCL collectives

[xyaohubery]

Summary

Diagnostic tool for detecting and localizing non-determinism in NCCL collective operations, addressing Issue #4.

Problem

Two identical training runs can diverge bitwise due to floating-point non-associativity in collective reductions (ring vs tree vs PAT algorithms, topology changes). This tool pinpoints the exact call and algorithm that introduces divergence.

What's Included

• bitwise_determinism_checker.py: Core engine with simulated Ring/Tree/PAT/NVLS algorithms, bitwise comparison framework, algorithm sweep, and actionable recommendations
• nccl_config_injector.py: NCCL config sweep, topology pinning, and torch.distributed integration

Usage

# Single-op diagnostic
python bitwise_determinism_checker.py --op allreduce --size 1M --runs 10

# Sweep across all algorithms
python bitwise_determinism_checker.py --sweep-algos --op allreduce --size 128M

# Full diagnostic suite
python bitwise_determinism_checker.py --full-suite --output report.json

Key Findings

• Ring + Simple: Deterministic (fixed rank order) — recommended for internode
• NVLS: Deterministic (hardware tree) — recommended for intranode with NVSwitch
• PAT: Potentially non-deterministic (atomic operations in shared memory)
• Tree: Non-deterministic without topology pinning (dynamic parent selection)

🤖 Generated with Claude Code