Hi, thanks for your great work and for sharing the resources!
I had a quick question regarding model tuning. Have you experimented with LoRA tuning on the synthetically longcontext generated data? If so, did you find it to be effective?
I'm currently trying to use the ProLong script you referenced, but it appears to require more than 8 A100 GPUs, which isn’t feasible for my setup at the moment. I’d appreciate any advice on running a lighter setup or alternative strategies that worked well for you.
Thanks again!
Hi, thanks for your great work and for sharing the resources!
I had a quick question regarding model tuning. Have you experimented with LoRA tuning on the synthetically longcontext generated data? If so, did you find it to be effective?
I'm currently trying to use the ProLong script you referenced, but it appears to require more than 8 A100 GPUs, which isn’t feasible for my setup at the moment. I’d appreciate any advice on running a lighter setup or alternative strategies that worked well for you.
Thanks again!