AdEMAMix32bit and PagedAdEMAMix32bit allocate a single-size state1 and fail on the first step

[caiotheodoro]

bnb.optim.AdEMAMix32bit and bnb.optim.PagedAdEMAMix32bit subclass Optimizer2State directly (bitsandbytes/optim/ademamix.py:355 and :385 on 8336490), so they never run AdEMAMix.init_state. That override allocates state1 as (2, *p.shape) to hold m1 and m2. The base Optimizer2State.init_state allocates p.shape. Every ademamix kernel then reads m2 from the second half of state1:

• CPU backend (_optimizer_update_32bit_cpu in bitsandbytes/backends/cpu/ops.py) indexes state1[0] and state1[1], so the first step() raises RuntimeError: output with shape [] doesn't match the broadcast shape [4096].
• Default backend (_optimizer_update_32bit in bitsandbytes/backends/default/ops.py, used on MPS) does the same indexing and raises the same error.
• CUDA (kOptimizer32bit2State in csrc/kernels.cu, line 674) loads m2 with Load(&(state1[n + i]), s3_vals, ...) on a buffer of n floats. I do not have a CUDA device to run this. From the source, it reads and later writes past the end of state1.

The scheduler is lost as well. Optimizer2State.update_step never applies t_alpha or t_beta3, so AdEMAMix32bit(t_alpha=100, t_beta3=100) runs unscheduled without any warning.

AdEMAMix8bit is not affected because it subclasses AdEMAMix. AdEMAMix(optim_bits=32) works, and that is what tests/test_optim.py constructs under the id "AdEMAMix32bit" (line 580), so the class itself has no test coverage. The class has had this parent since #1360 introduced it.

Repro, run on main at 8336490 with torch 2.14.0, Python 3.13.12, macOS arm64, pip install -e . (CPU build). Same output with dev = "mps".

import torch, bitsandbytes as bnb
dev = "cpu"
for cls in (bnb.optim.AdEMAMix32bit, bnb.optim.PagedAdEMAMix32bit, bnb.optim.AdEMAMix, bnb.optim.AdEMAMix8bit):
    p = torch.nn.Parameter(torch.randn(4096, device=dev))
    opt = cls([p])
    p.grad = torch.randn_like(p)
    try:
        opt.step()
        print(f"{cls.__name__:22s} OK   state1.shape={tuple(opt.state[p]['state1'].shape)}")
    except Exception as e:
        print(f"{cls.__name__:22s} FAIL state1.shape={tuple(opt.state[p]['state1'].shape)} {type(e).__name__}: {e}")

AdEMAMix32bit          FAIL state1.shape=(4096,) RuntimeError: output with shape [] doesn't match the broadcast shape [4096]
PagedAdEMAMix32bit     FAIL state1.shape=(4096,) RuntimeError: output with shape [] doesn't match the broadcast shape [4096]
AdEMAMix               OK   state1.shape=(2, 4096)
AdEMAMix8bit           OK   state1.shape=(2, 4096)

Fix: make AdEMAMix32bit subclass AdEMAMix with optim_bits=32, the same way AdEMAMix8bit does. PagedAdEMAMix32bit already subclasses AdEMAMix32bit and picks it up. PR follows.