Is the following a good method for determining learning rate in a neural network?

Question

Start with a default Learning rate of 0.1

Run the gradient descents normally.

If the cost function ever gives a higher value than the previous iteration, divide the learning rate by 10.

Repeat steps 1-3 until the learning rate becomes $10^{-6}$.

10xAI · Answer

Starting with a high value is not a good idea. Since we are not aware of where the training starts on the Loss surface, so it can easily diverge.
Also, in between we need a higher LR to quickly move on saddle points (plateaus).
 Both start and end should be at a small value
A well-known approach is called Cyclical-Learning-Rate suggested by Leslie N. Smith in this paper Link
It suggests following a Cyclic path e.g. Triangle, Sinusoidal, etc.
It says to first increase from a minimum bound and starts decreasing when reached a maximum bound.
$hspace{3cm}$
Paper has clearly explained -
- How to find the bounds
- How to decide the stepsize

Is the following a good method for determining learning rate in a neural network?

One Answer

Add your own answers!

Ask a Question