Need Help with Fine-Tuning LoRa Using Diffusion Pipe for Hyperrealistic Object Videos

[AlexNistiriuc]

I’m experiencing difficulties fine-tuning LoRa using Diffusion Pipe to generate hyperrealistic videos of objects. Despite trying various datasets, adjusting the training steps, and experimenting with the training duration, I haven't been able to achieve satisfactory results.

For instance, one specific challenge I face is ensuring that text, such as the title on a book cover, appears clear and accurate in the output. So far, the results are inconsistent, and I’m not sure what adjustments I should make to improve the quality.

Do you have any suggestions for settings, best practices, or procedures that could help me fine-tune LoRa more effectively in this context? Any guidance would be greatly appreciated.

Thank you!

[Srabontideb]

I have some suggessions:

1. Set equal to rank (e.g., rank=32, alpha=32) for stable training.
2. 1e-4 is a safe start; lower to 5e-5 if you see artifacts or overfitting.
3. 500–2000 steps depending on dataset size; use validation images to gauge.
4. cosine with warmup (100–200 steps) is generally reliable