Effect of Depth and Width on Local Minima in Deep Learning

Abstract

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.

References in corpus (14)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• On the Number of Linear Regions of Deep Neural Networks
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• The loss surface of deep and wide neural networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Benefits of depth in neural networks
• Bayesian Optimization with Exponential Convergence
• On the Computational Efficiency of Training Neural Networks
• Diverse Neural Network Learns True Target Functions
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Optimization Landscape and Expressivity of Deep CNNs
• Are ResNets Provably Better than Linear Predictors?

Cited by in corpus (11)

• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• Learning Constitutive Relations using Symmetric Positive Definite Neural Networks
• A Transfer Learning Evaluation of Deep Neural Networks for Image Classification
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
• Deep frequency principle towards understanding why deeper learning is faster
• Multi-Point Bandit Algorithms for Nonstationary Online Nonconvex Optimization
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks