Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

Abstract

An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data. We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other. Thus changes to the network parameters during training are biased towards those that (locally) simultaneously benefit many examples when such similarity exists. We support this hypothesis with heuristic arguments and perturbative experiments and outline how this can explain several common empirical observations about Deep Learning. Furthermore, our analysis is not just descriptive, but prescriptive. It suggests a natural modification to gradient descent that can greatly reduce overfitting.

To appear in ICLR 2020

References in corpus (10)

• Deep Learning with Differential Privacy
• Reconciling modern machine learning practice and the bias-variance trade-off
• The Loss Surfaces of Multilayer Networks
• Deep Learning is Robust to Massive Label Noise
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• On the Spectral Bias of Neural Networks
• Uniform convergence may be unable to explain generalization in deep learning
• Stiffness: A New Perspective on Generalization in Neural Networks
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Cited by in corpus (11)

• Learning with Instance-Dependent Label Noise: A Sample Sieve Approach
• Deep Learning Through the Lens of Example Difficulty
• Implicit competitive regularization in GANs
• Normalized Attention Without Probability Cage
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Stochastic Training is Not Necessary for Generalization
• Implicit Regularization via Neural Feature Alignment
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Extreme Memorization via Scale of Initialization
• Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignment