Skip to content

Implement Linear and Exponential Learning Rate Decay #5

Description

@tmr-monolith

The learning rate is very important and it would be much better to have more control over the learning rate decay.

Implement Linear and Exponential learning rate decay to get a good initial functionality improvement with this mechanism.

Linear Learning rate decay:

alpha_t = alpha_start - (alpha_start - alpha_end) * t / N

Here t is a variable based on the number of iterations, N is the total number of iterations, alpha_start is the initial learning rate, and alpha_end is the final learning rate after N iterations.

Exponential Learning Rate decay:

alpha_t = alpha * delta_t
delta_t = alpha_end / alpha_start ^ (1 / N - 1)) 

Implementation changes:

Learning rate decay should be implemented into the Optimiser.
The Optimiser class will need to take a final_learning_rate parameter. Then two functions will be required:

setupDecay(); // initialised the learning rate based on the type of learning decay required
progressDecay(); // progress the learning rate decay at the end of each epoch. 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions