Understanding Generalization through Visualizations

Abstract

The power of neural networks lies in their ability to generalize to unseen data, yet the underlying reasons for this phenomenon remain elusive. Numerous rigorous attempts have been made to explain generalization, but available bounds are still quite loose, and analysis does not always lead to true understanding. The goal of this work is to make generalization more intuitive. Using visualization methods, we discuss the mystery of generalization, the geometry of loss landscapes, and how the curse (or, rather, the blessing) of dimensionality causes optimizers to settle into minima that generalize well.

arXiv comment8 pages (excluding acknowledgments and references), 8 figures

References in corpus (15)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Large Batch Training of Convolutional Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• On the importance of single directions for generalization
• Averaging Weights Leads to Wider Optima and Better Generalization
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets
• Sharp Minima Can Generalize For Deep Nets
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Norm-Based Capacity Control in Neural Networks
• Data-Dependent Stability of Stochastic Gradient Descent
• Identifying Generalization Properties in Neural Networks
• Generalization Error of Invariant Classifiers
• On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularization
• Fast Rates for Empirical Risk Minimization of Strict Saddle Problems
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

Cited by in corpus (17)

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• The Case for Bayesian Deep Learning
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• The Intrinsic Dimension of Images and Its Impact on Learning
• Learning Optimal Representations with the Decodable Information Bottleneck
• Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
• Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Stochastic Training is Not Necessary for Generalization
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Divergence Regulated Encoder Network for Joint Dimensionality Reduction and Classification
• Deforming the Loss Surface to Affect the Behaviour of the Optimizer
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• The Uncanny Similarity of Recurrence and Depth
• Active Learning at the ImageNet Scale
• Deforming the Loss Surface