SGD on Neural Networks Learns Functions of Increasing Complexity

Abstract

We perform an experimental study of the dynamics of Stochastic Gradient Descent (SGD) in learning deep neural networks for several real and synthetic classification tasks. We show that in the initial epochs, almost all of the performance improvement of the classifier obtained by SGD can be explained by a linear classifier. More generally, we give evidence for the hypothesis that, as iterations progress, SGD learns functions of increasing complexity. This hypothesis can be helpful in explaining why SGD-learned classifiers tend to generalize well even in the over-parameterized regime. We also show that the linear classifier learned in the initial stages is "retained" throughout the execution even if training is continued to the point of zero training error, and complement this with a theoretical result in a simplified model. Key to our work is a new measure of how well one classifier explains the performance of another, based on conditional mutual information.

arXiv commentSubmitted to NeurIPS 2019

References in corpus (4)

• Opening the Black Box of Deep Neural Networks via Information
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Data-Dependent Stability of Stochastic Gradient Descent

Cited by in corpus (25)

• Shortcut Learning in Deep Neural Networks
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• The Pitfalls of Simplicity Bias in Neural Networks
• Towards Understanding the Spectral Bias of Deep Learning
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Understanding and Improving Fast Adversarial Training
• Gradient Starvation: A Learning Proclivity in Neural Networks
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learning
• What Makes Training Multi-Modal Classification Networks Hard?
• Understanding Deep Learning via Decision Boundary
• Learning Not to Learn in the Presence of Noisy Labels
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization
• The Low-Rank Simplicity Bias in Deep Networks
• Study of nonlinear optical diffraction patterns using machine learning models based on ResNet 152 architecture
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Deep frequency principle towards understanding why deeper learning is faster
• Learning time-scales in two-layers neural networks
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• Dropout as a Regularizer of Interaction Effects
• Generalization by design: Shortcuts to Generalization in Deep Learning