Predicting the Generalization Gap in Deep Networks with Margin Distributions

Abstract

As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum). Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

arXiv commentPublished in ICLR 2019

References in corpus (10)

• Understanding deep learning requires rethinking generalization
• Manifold Mixup: Better Representations by Interpolating Hidden States
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Robust Large Margin Deep Neural Networks
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Practical Black-Box Attacks against Machine Learning
• Stronger generalization bounds for deep nets via a compression approach
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• A Surprising Linear Relationship Predicts Test Performance in Deep Networks
• Large Margin Deep Networks for Classification

Cited by in corpus (42)

• Fantastic Generalization Measures and Where to Find Them
• The Modern Mathematics of Deep Learning
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Predicting Neural Network Accuracy from Weights
• Exploring the Limits of Large Scale Pre-training
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
• Learning Optimal Representations with the Decodable Information Bottleneck
• Assessing Generalization of SGD via Disagreement
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• Membership Inference Attacks and Generalization: A Causal Perspective
• Representation Based Complexity Measures for Predicting Generalization in Deep Learning
• Potential adversarial samples for white-box attacks
• Directional convergence and alignment in deep learning
• Generalization bounds for deep learning
• Hold me tight! Influence of discriminative features on deep network boundaries
• Towards Task and Architecture-Independent Generalization Gap Predictors
• Label-Free Model Evaluation with Semi-Structured Dataset Representations
• Investigating Decision Boundaries of Trained Neural Networks
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Neural Complexity Measures
• Information-Theoretic Local Minima Characterization and Regularization
• In Search of Robust Measures of Generalization
• Hyperplane Arrangements of Trained ConvNets Are Biased
• Why flatness does and does not correlate with generalization for deep neural networks
• Abstraction Mechanisms Predict Generalization in Deep Neural Networks
• Ranking Neural Checkpoints
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• On Predicting Generalization using GANs
• Gi and Pal Scores: Deep Neural Network Generalization Statistics
• Explaining generalization in deep learning: progress and fundamental limits
• The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs
• Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• The role of invariance in spectral complexity-based generalization bounds
• Noisy Feature Mixup
• Predicting Generalization in Deep Learning via Local Measures of Distortion
• Fractal Dimension Generalization Measure
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• DNNs as Layers of Cooperating Classifiers
• An Investigation of the Weight Space to Monitor the Training Progress of Neural Networks