Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology

Abstract

Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with $N$ ($N$ being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width $N$ following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used LeCun's initialization and obtain an over-parameterization requirement for the single wide layer of order $N^2.$

arXiv commentAccepted at NeurIPS 2020

References in corpus (25)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• The loss surface of deep and wide neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Optimization Landscape and Expressivity of Deep CNNs
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• Sharp interpolation inequalities on the sphere : new methods and consequences
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Small nonlinearities in activation functions create bad local minima in neural networks
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Neural Networks Learning and Memorization with (almost) no Over-Parameterization
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
• Memory capacity of neural networks with threshold and ReLU activations
• On the infinite width limit of neural networks with a standard parameterization

Cited by in corpus (1)

• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime