The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent

Abstract

This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But when gradient confusion is low, data samples interact harmoniously, and training proceeds quickly. Through theoretical and experimental results, we demonstrate how the neural network architecture affects gradient confusion, and thus the efficiency of training. Our results show that, for popular initialization techniques, increasing the width of neural networks leads to lower gradient confusion, and thus faster model training. On the other hand, increasing the depth of neural networks has the opposite effect. Our results indicate that alternate initialization techniques or networks using both batch normalization and skip connections help reduce the training burden of very deep networks.

arXiv commentICML 2020 camera-ready version

References in corpus (38)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Improving Generalization Performance by Switching from Adam to SGD
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• How Does Batch Normalization Help Optimization?
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• All you need is a good init
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Data-dependent Initializations of Convolutional Neural Networks
• The loss surface of deep and wide neural networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The Singular Values of Convolutional Layers
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Fixup Initialization: Residual Learning Without Normalization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Benefits of depth in neural networks
• Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Generalization in Deep Networks: The Role of Distance from Initialization
• On exponential convergence of SGD in non-convex over-parametrized learning
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• A Mean Field Theory of Batch Normalization
• Stiffness: A New Perspective on Generalization in Neural Networks
• The loss landscape of overparameterized neural networks
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron
• Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
• The Effect of Network Width on the Performance of Large-batch Training

Cited by in corpus (19)

• MetaPoison: Practical General-purpose Clean-label Data Poisoning
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Better Theory for SGD in the Nonconvex World
• The learnability scaling of quantum states: restricted Boltzmann machines
• Strong error analysis for stochastic gradient descent optimization algorithms
• ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Rethinking Bottleneck Structure for Efficient Mobile Network Design
• Hierarchical Neural Architecture Search via Operator Clustering
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• Which Minimizer Does My Neural Network Converge To?
• Extreme Memorization via Scale of Initialization
• Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
• Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignment
• Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• Inherent Noise in Gradient Based Methods