Which Minimizer Does My Neural Network Converge To?

Abstract

The loss surface of an overparameterized neural network (NN) possesses many global minima of zero training error. We explain how common variants of the standard NN training procedure change the minimizer obtained. First, we make explicit how the size of the initialization of a strongly overparameterized NN affects the minimizer and can deteriorate its final test performance. We propose a strategy to limit this effect. Then, we demonstrate that for adaptive optimization such as AdaGrad, the obtained minimizer generally differs from the gradient descent (GD) minimizer. This adaptive minimizer is changed further by stochastic mini-batch training, even though in the non-adaptive case, GD and stochastic GD result in essentially the same minimizer. Lastly, we explain that these effects remain relevant for less overparameterized NNs. While overparameterization has its benefits, our work highlights that it induces sources of error absent from underparameterized models.

References in corpus (19)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• On Exact Computation with an Infinitely Wide Neural Net
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• On the Spectral Bias of Neural Networks
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• On the Convergence Rate of Training Recurrent Neural Networks
• Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• A type of generalization error induced by initialization in deep neural networks
• Minimum weight norm models do not always generalize well for over-parameterized problems
• Which Minimizer Does My Neural Network Converge To?

Cited by in corpus (2)

• Fast and Sample-Efficient Interatomic Neural Network Potentials for Molecules and Materials Based on Gaussian Moments
• Which Minimizer Does My Neural Network Converge To?