Theory of Deep Learning IIb: Optimization Properties of SGD

Abstract

In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper is theoretical and experimental evidence for the following conjecture about SGD: SGD concentrates in probability -- like the classical Langevin equation -- on large volume, "flat" minima, selecting flat minimizers which are with very high probability also global minimizers

References in corpus (1)

• Theory II: Landscape of the Empirical Risk in Deep Learning

Cited by in corpus (19)

• Multi-Objective Loss Balancing for Physics-Informed Deep Learning
• Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Strength of Minibatch Noise in SGD
• Consistent Sparse Deep Learning: Theory and Computation
• Theory III: Dynamics and Generalization in Deep Networks
• On the Distributional Properties of Adaptive Gradients
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Global Convergence of SGD On Two Layer Neural Nets
• What training reveals about neural network complexity
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Relative Flatness and Generalization
• Volumization as a Natural Generalization of Weight Decay