A Deep Conditioning Treatment of Neural Networks
arXiv:2002.01523
Abstract
We study the role of depth in training randomly initialized overparameterized neural networks. We give a general result showing that depth improves trainability of neural networks by improving the conditioning of certain kernel matrices of the input data. This result holds for arbitrary non-linear activation functions under a certain normalization. We provide versions of the result that hold for training just the top layer of the neural network, as well as for training all layers, via the neural tangent kernel. As applications of these general results, we provide a generalization of the results of Das et al. (2019) showing that learnability of deep random neural networks with a large class of non-linear activations degrades exponentially with depth. We also show how benign overfitting can occur in deep neural networks via the results of Bartlett et al. (2019b). We also give experimental evidence that normalized versions of ReLU are a viable alternative to more complex operations like Batch Normalization in training deep neural networks.
arXiv commentIn proceedings of ALT 2021
References in corpus (33)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- Strictly and non-strictly positive definite functions on spheres
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Exponential expressivity in deep neural networks through transient chaos
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- Understanding Deep Neural Networks with Rectified Linear Units
- Learning One-hidden-layer Neural Networks with Landscape Design
- Fixup Initialization: Residual Learning Without Normalization
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
- The Shattered Gradients Problem: If resnets are the answer, then what is the question?
- Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
- Stronger generalization bounds for deep nets via a compression approach
- Norm-Based Capacity Control in Neural Networks
- SGD Learns the Conjugate Kernel Class of the Network
- Learning One Convolutional Layer with Overlapping Patches
- On the ability of neural nets to express distributions
- On the Expressive Efficiency of Sum Product Networks
- Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
- On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
- Depth Separation for Neural Networks
- Gradient Descent for One-Hidden-Layer Neural Networks: Polynomial Convergence and SQ Lower Bounds
- Learning Two Layer Rectified Neural Networks in Polynomial Time
- On the Learnability of Deep Random Networks