Integrate RAD's LPT recipes

[nfrumkin]

Integrate RAD's LPT recipes

Summary

Integrates AMD Research's (RAD) low-precision-training (LPT) recipes into ALTO on top of the existing MXFP4 stack. The core addition is AdaHOP — per-slot Hadamard mode selection driven by a calibration phase — wired in through the mxfp4_adahop scheme and vendored as the 3rdparty/adahop submodule (AMDResearch/low-precision-training, branch AdaHOP_C42). Alongside it this PR brings in the supporting recipes and tooling that RAD's runs depend on:

• AdaHOP modifier (alto/modifiers/lpt/adahop.py, adahop_internals/): a single MXFP4AdaHOPWrapper set at conversion time with all modes "none" (plain MXFP4), then flipped in place at the end of calibration — no wrapper-type swap and no new nn.Parameter, so optimizer state references stay valid across the Phase-A→Phase-B transition and across checkpoint resume. Isolated from AdaHOP's vendored torchtitan via alto/_adahop_bridge.py so it can never shadow ALTO's own torchtitan submodule.
• midmax scale selection (alto/kernels/fp4/mxfp4/mxfp_quantization.py): a general SCALE_SELECTION constexpr flag in the quantization kernel (replacing a dedicated midmax path), plus the uos/uos6/deosc/midmax recipe variants.
• M+Adam optimizer (alto/components/m_adam.py): a hybrid additive (AdamW mantissa) / multiplicative (Madam-style exponent) optimizer with independent LR/WD/schedule on the exponent, wired into torchtitan's config.optimizer.build(...) path, used for the de-oscillation (deosc) recipe.
• Forward-only MXFP4 (mxfp4_forward_only.py) and per-layer gradient clipping (alto/modifiers/lpt/grad_clip.py) modifiers.
• Debug/observer tooling (alto/modifiers/debug/): per-step tensor capture (DebugObserverModifier) and per-expert MoE outlier-pattern capture (MoEMatmulPatternObserverModifier) for the two Grouped GEMMs of gpt_oss MoE blocks, with visualizers under scripts/.
• Recipes & registry: new LPT recipe YAMLs and config_registry entries for gpt_oss_20b (mxfp4_base, adahop, lpt_deosc, lpt_midmax, lpt_uos, lpt_uos6, moe_pattern_obs) and llama3 (adahop, adahop_debug, hadamard, fwdonly).
• Plotting (plotting/plot_training_stats.py + .toml configs): unified loss / grad-norm plotting from slurm .out logs and/or TensorBoard event files.
• Infra: multinode training scripts + Dockerfile.multinode, RDMA test harness (rdma_tests/, RDMA.md), and run_id/TensorBoard support in alto/train.py.

Total: 84 files, ~9.1k insertions.

Validation

Unit tests — 84 new test functions, GPU-gated where they exercise kernels:

• AdaHOP (28): numerical parity (AdaHOP "none" mode matches plain MXFP4; per-mode grad_w bias and Hadamard forward error bounded vs the bf16 reference; SR reduces grad_w bias), wrapper lifecycle, bridge isolation, calibration hooks, modifier helpers, pattern aggregation.
• Debug/observer (43): observer hooks, debug-observer modifier, MoE pattern hooks, and gradient clipping.
• midmax (13): E2M1 midmax constant (= 7.0) and scale-bump correctness against a pure-Python reference on crafted amax inputs (bump iff amax_normalized > 7.0).

Numerical parity is checked against pure-PyTorch references (bf16 and plain-MXFP4), which keeps AdaHOP/kernel porting errors separable from MXFP4 quantization error. AdaHOP "none" is asserted to reproduce the plain-MXFP4 baseline exactly.

End-to-end: gpt_oss_20b MXFP4 training runs with the adahop / deosc / midmax recipes vs the mxfp4_base baseline.

• Baseline (mxfp4_base) final loss: <TODO>
• AdaHOP final loss: <TODO>
• de-osc final loss: <TODO>
• (loss/grad-norm curves via plotting/plot_training_stats.py)

Integration smoke scripts added under tests/integration/ (llama3_debugmodel_baseline, llama3_debugmodel_adahop, llama3_debugmodel_adahop_short, llama3_1b_adahop) for a quick recipe-runs check.

Scope

• AdaHOP is exposed only through the mxfp4_adahop scheme; full_precision_backward remains restricted to the plain mxfp4 scheme.
• The observed step for MoEMatmulPatternObserverModifier must run in eager mode (the torch._grouped_mm monkeypatch is invisible to a compiled graph); the low-precision path itself is backend-agnostic (CDNA3 loop fallback and CDNA4 kernels alike).
• MXFP4ForwardOnlyLinearFunction supports the non-CDNA4 (QDQ) path only (asserted at entry); midmax/SCALE_SELECTION kernel paths use the CDNA4 (gfx950) ASM path where available.
• GradientClippingModifier must be listed after LowPrecisionTrainingModifier in a recipe so swap_params has already tagged the wrapped weights.
• The 3rdparty/adahop submodule is loaded by file path (never added to sys.modules as a package) specifically so its vendored torchtitan cannot shadow ALTO's.

Reviewer Notes

• AdaHOP in-place mode flip is the load-bearing design decision. Calibration starts every slot at mode "none" and mutates modes in place at the Phase-A→Phase-B boundary rather than swapping wrapper types or allocating new parameters — this is what keeps optimizer state and checkpoint resume valid. Resume logic in on_pre_step inspects the checkpointed CalibrationStateManager: if calibration already completed it re-applies restored modes and skips calibration (clean resume).
• Submodule required. This PR adds 3rdparty/adahop (AMDResearch/low-precision-training, branch AdaHOP_C42) — reviewers/CI must git submodule update --init --recursive.
• midmax is now a general flag, not a dedicated code path: the kernel selects behavior via a SCALE_SELECTION constexpr, which also carries the uos/uos6 variants.
• Merge history: rad/lpt accumulates work from rad/midmax, rad/scale-selection, and yann_moe (M+Adam / de-oscillation). The final commits repoint the adahop/lpt recipe configs back to the original rad/lpt baseline files.
• Debug/observer modifiers and the plotting/RDMA/multinode tooling are analysis and infra utilities — they do not touch the default training path unless explicitly listed in a recipe.

[hann-wang]

Hi @nfrumkin, I cannot access AdaHOP. It does not seem to be a public repo.

[hann-wang]

The rounding does not work this way. The 0.25 value_to_add means that we will increase the exponent by 1 if the matissa is larger than 1.75.

For $123.2=2^6*1.925$, the exponent is 6 without value_to_add, and 7 with value_to_add.