Gi and Pal Scores: Deep Neural Network Generalization Statistics
arXiv:2104.03469
Abstract
The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of regression, classification, and control tasks. However, despite these successes, the field lacks strong theoretical error bounds and consistent measures of network generalization and learned invariances. In this work, we introduce two new measures, the Gi-score and Pal-score, that capture a deep neural network's generalization capabilities. Inspired by the Gini coefficient and Palma ratio, measures of income inequality, our statistics are robust measures of a network's invariance to perturbations that accurately predict generalization gaps, i.e., the difference between accuracy on training and test sets.
arXiv commentAccepted to RobustML Workshop at ICLR 2021
References in corpus (6)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- mixup: Beyond Empirical Risk Minimization
- Manifold Mixup: Better Representations by Interpolating Hidden States
- Predicting the Generalization Gap in Deep Networks with Margin Distributions
- Representation Based Complexity Measures for Predicting Generalization in Deep Learning
- Robustness to Augmentations as a Generalization metric