Thank you for the nice compact work.
We have started recently to face an ambiguous error casing the loss to become nan during the training. After enabling anomaly detection " torch.autograd.set_detect_anomaly(True)"
We got this:
UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:
...stacktrace...
.venv/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 569, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
...stacktrace...
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'MmBackward0' returned nan values in its 1th output.
Could it be caused by some numerical instability (nan or inf)?
Thank you for the nice compact work.
We have started recently to face an ambiguous error casing the loss to become nan during the training. After enabling anomaly detection "
torch.autograd.set_detect_anomaly(True)"We got this:
UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:
...stacktrace...
.venv/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 569, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
...stacktrace...
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'MmBackward0' returned nan values in its 1th output.
Could it be caused by some numerical instability (nan or inf)?