The learning rate is very important and it would be much better to have more control over the learning rate decay.
Implement Linear and Exponential learning rate decay to get a good initial functionality improvement with this mechanism.
Linear Learning rate decay:
alpha_t = alpha_start - (alpha_start - alpha_end) * t / N
Here t is a variable based on the number of iterations, N is the total number of iterations, alpha_start is the initial learning rate, and alpha_end is the final learning rate after N iterations.
Exponential Learning Rate decay:
alpha_t = alpha * delta_t
delta_t = alpha_end / alpha_start ^ (1 / N - 1))
Implementation changes:
Learning rate decay should be implemented into the Optimiser.
The Optimiser class will need to take a final_learning_rate parameter. Then two functions will be required:
setupDecay(); // initialised the learning rate based on the type of learning decay required
progressDecay(); // progress the learning rate decay at the end of each epoch.
The learning rate is very important and it would be much better to have more control over the learning rate decay.
Implement Linear and Exponential learning rate decay to get a good initial functionality improvement with this mechanism.
Linear Learning rate decay:
Here
tis a variable based on the number of iterations, N is the total number of iterations,alpha_startis the initial learning rate, andalpha_endis the final learning rate after N iterations.Exponential Learning Rate decay:
Implementation changes:
Learning rate decay should be implemented into the Optimiser.
The Optimiser class will need to take a final_learning_rate parameter. Then two functions will be required: