I found a bug in your code.
When i register a hook to lora_A tensor, I found that its backward times will increase.
Which means that, in batch n, the output of lora block will backward its grad to all the lora pair before, the hook said that, the output of lora backward to "from lora in batch n to lora in batch 0". And then , only one backward to the input of lora block. But surprisely, the memory doesn't increase. I wonder why it happened. I think it might have something is cumulate cross the batch, and its grad is crossing the lora pair.
Sorry for my poor English, and if you didn't get my mean, please tell me.Thank you !
I found a bug in your code.
When i register a hook to lora_A tensor, I found that its backward times will increase.
Which means that, in batch n, the output of lora block will backward its grad to all the lora pair before, the hook said that, the output of lora backward to "from lora in batch n to lora in batch 0". And then , only one backward to the input of lora block. But surprisely, the memory doesn't increase. I wonder why it happened. I think it might have something is cumulate cross the batch, and its grad is crossing the lora pair.
Sorry for my poor English, and if you didn't get my mean, please tell me.Thank you !