A Comparison of Optimization Algorithms for Deep Learning

Abstract

In recent years, we have witnessed the rise of deep learning. Deep neural networks have proved their success in many areas. However, the optimization of these networks has become more difficult as neural networks going deeper and datasets becoming bigger. Therefore, more advanced optimization algorithms have been proposed over the past years. In this study, widely used optimization algorithms for deep learning are examined in detail. To this end, these algorithms called adaptive gradient methods are implemented for both supervised and unsupervised tasks. The behaviour of the algorithms during training and results on four image datasets, namely, MNIST, CIFAR-10, Kaggle Flowers and Labeled Faces in the Wild are compared by pointing out their differences against basic optimization algorithms.

International Journal of Pattern Recognition and Artificial Intelligence, 2020

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• On the Convergence of Adam and Beyond
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates

Cited by in corpus (7)

• Prediction of lung and colon cancer through analysis of histopathological images by utilizing Pre-trained CNN models with visualization of class activation and saliency maps
• Application of multilayer perceptron with data augmentation in nuclear physics
• Unveiling The Factors of Aesthetic Preferences with Explainable AI
• Stochastic Modeling of Inhomogeneities in the Aortic Wall and Uncertainty Quantification using a Bayesian Encoder-Decoder Surrogate
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• An approximate Riemann solver approach in Physics-Informed Neural Networks for hyperbolic conservation laws
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning