Optimization for deep learning: theory and algorithms

Abstract

When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.

38 pages of main body; 5 pages of appendix; 12 pages of references

References in corpus (96)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• ADADELTA: An Adaptive Learning Rate Method
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• An overview of gradient descent optimization algorithms
• Neural Architecture Search with Reinforcement Learning
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On the Convergence of Adam and Beyond
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Improving Generalization Performance by Switching from Adam to SGD
• Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
• Guaranteed Matrix Completion via Non-convex Factorization
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• All you need is a good init
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• A Universal Catalyst for First-Order Optimization
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Stabilizing the Lottery Ticket Hypothesis
• Sharp Minima Can Generalize For Deep Nets
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Barzilai-Borwein Step Size for Stochastic Gradient Descent
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Learning One-hidden-layer Neural Networks with Landscape Design
• Fixup Initialization: Residual Learning Without Normalization
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Gradient descent aligns the layers of deep linear networks
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Image Classification at Supercomputer Scale
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• The Emergence of Spectral Universality in Deep Networks
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Mathematics of Deep Learning
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• Depth Creates No Bad Local Minima
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Global optimality conditions for deep neural networks
• Jamming in multilayer supervised learning models
• Accelerating SGD with momentum for over-parameterized learning
• Theory II: Landscape of the Empirical Risk in Deep Learning
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Adding One Neuron Can Eliminate All Bad Local Minima
• Elimination of All Bad Local Minima in Deep Learning
• A mean-field limit for certain deep neural networks
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Global Convergence of Block Coordinate Descent in Deep Learning
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Theoretical properties of the global optimizer of two layer neural network
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Porcupine Neural Networks: (Almost) All Local Optima are Global
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function
• Depth creates no more spurious local minima
• The Multilinear Structure of ReLU Networks
• Positively Scale-Invariant Flatness of ReLU Neural Networks
• On Connected Sublevel Sets in Deep Learning
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization

Cited by in corpus (42)

• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Facial Emotion Recognition: State of the Art Performance on FER2013
• Early Melanoma Diagnosis with Sequential Dermoscopic Images
• Better Theory for SGD in the Nonconvex World
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• The Practicality of Stochastic Optimization in Imaging Inverse Problems
• Adaptive neighborhood Metric learning
• Random Reshuffling: Simple Analysis with Vast Improvements
• An Improved Analysis of Stochastic Gradient Descent with Momentum
• The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent
• From Symmetry to Geometry: Tractable Nonconvex Problems
• Recent advances in deep learning theory
• On the distance between two neural networks and the stability of learning
• Deep learning for solution and inversion of structural mechanics and vibrations
• MARINA: Faster Non-Convex Distributed Learning with Compression
• LaProp: Separating Momentum and Adaptivity in Adam
• On Learning Rates and Schrödinger Operators
• Memory capacity of neural networks with threshold and ReLU activations
• Bounded nonlinear forecasts of partially observed geophysical systems with physics-constrained deep learning
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• The dilemma of quantum neural networks
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Model Complexity of Deep Learning: A Survey
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Learning by Turning: Neural Architecture Aware Optimisation
• Sketchy Empirical Natural Gradient Methods for Deep Learning
• On the Distributional Properties of Adaptive Gradients
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Accelerating variational quantum algorithms with multiple quantum processors
• The loss landscape of deep linear neural networks: a second-order analysis
• NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning
• Momentum Centering and Asynchronous Update for Adaptive Gradient Methods
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Advancing biological super-resolution microscopy through deep learning: a brief review
• A deep neural network for multi-species fish detection using multiple acoustic cameras
• Adaptive Hierarchical Hyper-gradient Descent
• A Probabilistic Approach to Neural Network Pruning
• Generalization Bounds for High-dimensional M-estimation under Sparsity Constraint