Understanding Generalization through Visualizations

Abstract

The power of neural networks lies in their ability to generalize to unseen data, yet the underlying reasons for this phenomenon remain elusive. Numerous rigorous attempts have been made to explain generalization, but available bounds are still quite loose, and analysis does not always lead to true understanding. The goal of this work is to make generalization more intuitive. Using visualization methods, we discuss the mystery of generalization, the geometry of loss landscapes, and how the curse (or, rather, the blessing) of dimensionality causes optimizers to settle into minima that generalize well.

8 pages (excluding acknowledgments and references), 8 figures

References in corpus (5)

• Large Batch Training of Convolutional Networks
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets
• Norm-Based Capacity Control in Neural Networks
• Identifying Generalization Properties in Neural Networks
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

Cited by in corpus (17)

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• The Case for Bayesian Deep Learning
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• The Intrinsic Dimension of Images and Its Impact on Learning
• Learning Optimal Representations with the Decodable Information Bottleneck
• Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
• Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Stochastic Training is Not Necessary for Generalization
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Divergence Regulated Encoder Network for Joint Dimensionality Reduction and Classification
• Deforming the Loss Surface to Affect the Behaviour of the Optimizer
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• The Uncanny Similarity of Recurrence and Depth
• Active Learning at the ImageNet Scale
• Deforming the Loss Surface