Why not merge BA during training?

[rp7sv]

During training, the low-rank matrices create a bypass, which introduces additional latency. So why not merge the low-rank weights directly into W during training, just like we do at inference time? I'm not sure whether this would affect gradient flow？

[XuRan-Zhao]

During training, the low-rank matrices create a bypass, which introduces additional latency. So why not merge the low-rank weights directly into W during training, just like we do at inference time? I'm not sure whether this would affect gradient flow？

May I ask if you know the answer to this question now? Can you clarify it? I also have the same doubts.

[jim-min]

But then you cannot freeze the W, which is the reason we use LoRA?

[SadMadLad]

I think if you are doing different downstream tasks, merging is then not straightforward as you would need to switch between different sets of weights then. Somewhere in the paper I saw:

"LoRA also has its limitations. For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency. Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical."