Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

Abstract

It is widely observed that deep learning models with learned parameters generalize well, even with much more model parameters than the number of training samples. We systematically investigate the underlying reasons why deep neural networks often generalize well, and reveal the difference between the minima (with the same training error) that generalize well and those they don't. We show that it is the characteristics the landscape of the loss function that explains the good generalization capability. For the landscape of loss function for deep networks, the volume of basin of attraction of good minima dominates over that of poor minima, which guarantees optimization methods with random initialization to converge to good minima. We theoretically justify our findings through analyzing 2-layer neural networks; and show that the low-complexity solutions have a small norm of Hessian matrix with respect to model parameters. For deeper networks, extensive numerical evidence helps to support our arguments.

References in corpus (4)

• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

Cited by in corpus (62)

• Shortcut Learning in Deep Neural Networks
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Three Factors Influencing Minima in SGD
• Generalization in Deep Learning
• Optimization for deep learning: theory and algorithms
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Overfitting Mechanism and Avoidance in Deep Neural Networks
• Review: Deep Learning in Electron Microscopy
• DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Universal Effectiveness of High-Depth Circuits in Variational Eigenproblems
• Training behavior of deep neural network in frequency domain
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Dimensionality compression and expansion in Deep Neural Networks
• Understanding training and generalization in deep learning by Fourier analysis
• The loss landscape of overparameterized neural networks
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• On the different regimes of Stochastic Gradient Descent
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Gradient Descent Quantizes ReLU Network Features
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Generalization in Machine Learning via Analytical Learning Theory
• Generalization bounds for deep learning
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• Neural networks are a priori biased towards Boolean functions with low entropy
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Laziness, Barren Plateau, and Noise in Machine Learning
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• Is SGD a Bayesian sampler? Well, almost
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• Emergent Computations in Trained Artificial Neural Networks and Real Brains
• Stochastic Training is Not Necessary for Generalization
• The asymptotic spectrum of the Hessian of DNN throughout training
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Understanding Deep Learning Generalization by Maximum Entropy
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Information-Theoretic Local Minima Characterization and Regularization
• Understanding over-parameterized deep networks by geometrization
• Why flatness does and does not correlate with generalization for deep neural networks
• Flatness is a False Friend
• Deforming the Loss Surface to Affect the Behaviour of the Optimizer
• On the achievability of blind source separation for high-dimensional nonlinear source mixtures
• Consensus-based Interpretable Deep Neural Networks with Application to Mortality Prediction
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Deep network as memory space: complexity, generalization, disentangled representation and interpretability
• Convolutional Gated MLP: Combining Convolutions & gMLP
• AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations
• Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
• Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment
• Deforming the Loss Surface
• Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization
• Good Classifiers are Abundant in the Interpolating Regime
• On Large Batch Training and Sharp Minima: A Fokker-Planck Perspective
• Minimum sharpness: Scale-invariant parameter-robustness of neural networks
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks