Exploring Generalization in Deep Learning

Abstract

With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.

19 pages, 8 figures

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Generalization Error of Invariant Classifiers
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Cited by in corpus (48)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation
• Neural Lander: Stable Drone Landing Control using Learned Dynamics
• A deep learning-based remaining useful life prediction approach for bearings
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• The Role of Neural Network Activation Functions
• The Modern Mathematics of Deep Learning
• Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Review: Deep Learning in Electron Microscopy
• The no-free-lunch theorems of supervised learning
• Generalisation error in learning with random features and the hidden manifold model
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• On the different regimes of Stochastic Gradient Descent
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• On the training dynamics of deep networks with $L_2$ regularization
• Uncertainty-based Modulation for Lifelong Learning
• Deep Learning in current Neuroimaging: a multivariate approach with power and type I error control but arguable generalization ability
• Reducing Communication for Split Learning by Randomized Top-k Sparsification
• Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
• Exploring the Function Space of Deep-Learning Machines
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• Guided Unsupervised Learning by Subaperture Decomposition for Ocean SAR Image Retrieval
• Gaussian Universality of Perceptrons with Random Labels
• Effects of quantum resources on the statistical complexity of quantum circuits
• Fix your classifier: the marginal value of training the last weight layer
• Influence-guided Data Augmentation for Neural Tensor Completion
• Perturbated Gradients Updating within Unit Space for Deep Learning
• A Shooting Formulation of Deep Learning
• On generalization bounds for deep networks based on loss surface implicit regularization
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
• Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
• On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
• The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs
• Leveraging Local Variation in Data: Sampling and Weighting Schemes for Supervised Deep Learning
• Auditing Robot Learning for Safety and Compliance during Deployment
• Robust Generalization of Quadratic Neural Networks via Function Identification
• MOFHEI: Model Optimizing Framework for Fast and Efficient Homomorphically Encrypted Neural Network Inference
• Noise Stability Regularization for Improving BERT Fine-tuning
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics
• BN-invariant sharpness regularizes the training model to better generalization