Guidelines for selecting an optimizer for training neural networks

According to Kingma and Ba (2014) Adam has been developed for "large datasets and/or high-dimensional parameter spaces".

The authors claim that: "[Adam] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives" (page 9).

In the paper, there are some simulations where Adam is compared to SGDNesterov, AdaGrad, RMSProp (MNIST, IMDB, page 6; CIFAR10 (ConvNet), page 7). Adam does very well compared to the others. The authors find that Adam converges faster than AdaGrad in a convolutional network (5x5 convolution filters, 3x3 max pooling with stride of 2 that are followed by a fully connected layer of 1000 rectified linear hidden units).

Overall, Adam appears to be a good choice to start with for most (non-shallow) problems.

Nadam(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, schedule_decay=0.004)

Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum.

1. AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse data such as tf-idf, etc. Adagrad can be useful.

2. AdaDelta, RMSProp almost works on similar lines with the only difference in Adadelta you don't require an initial learning rate constant to start with.

3. Adam combines the good properties of Adadelta and RMSprop and hence tend to do better for most of the problems.

4. Stochastic gradient descent is very basic and is seldom used now. One problem is with the global learning rate associated with the same. Hence it doesn't work well when the parameters are in different scales since a low learning rate will make the learning slow while a large learning rate might lead to oscillations. Also Stochastic gradient descent generally has a hard time escaping the saddle points. Adagrad, Adadelta, RMSprop, and ADAM generally handle saddle points better. SGD with momentum renders some speed to the optimization and also helps escape local minima better.

My personal approach is to pick the optimizer that is newest (i.e. newest-published-in-a-peer-reviewed-journal), because they usually report results on standard datasets, or beat state of the art, or both. When I use Caffe for example, I always use Adam.