A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning

Abstract

One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient descent. But there are some problems with regular gradient descent ie. it is quite slow and not that accurate. This article aims to give an introduction to optimization strategies to gradient descent. In addition, we shall also discuss the architecture of these algorithms and further optimization of Neural Networks in general

References in corpus (12)

• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• The Loss Surfaces of Multilayer Networks
• Shake-Shake regularization
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• Efficient approaches for escaping higher order saddle points in non-convex optimization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Adding One Neuron Can Eliminate All Bad Local Minima
• Elimination of All Bad Local Minima in Deep Learning
• Stochastic gradient descent with random learning rate
• Transcendental Functions with a Complex Twist