Representation Based Complexity Measures for Predicting Generalization in Deep Learning

Abstract

Deep Neural Networks can generalize despite being significantly overparametrized. Recent research has tried to examine this phenomenon from various view points and to provide bounds on the generalization error or measures predictive of the generalization gap based on these viewpoints, such as norm-based, PAC-Bayes based, and margin-based analysis. In this work, we provide an interpretation of generalization from the perspective of quality of internal representations of deep neural networks, based on neuroscientific theories of how the human visual system creates invariant and untangled object representations. Instead of providing theoretical bounds, we demonstrate practical complexity measures which can be computed ad-hoc to uncover generalization behaviour in deep models. We also provide a detailed description of our solution that won the NeurIPS competition on Predicting Generalization in Deep Learning held at NeurIPS 2020. An implementation of our solution is available at https://github.com/parthnatekar/pgdl.

Winning Solution of the NeurIPS 2020 Competition on Predicting Generalization in Deep Learning

References in corpus (6)

• mixup: Beyond Empirical Risk Minimization
• Manifold Mixup: Better Representations by Interpolating Hidden States
• On the importance of single directions for generalization
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Predicting the Generalization Gap in Deep Networks with Margin Distributions

Cited by in corpus (7)

• Homophily-oriented Heterogeneous Graph Rewiring
• Assessing Generalization of SGD via Disagreement
• Measuring Generalization with Optimal Transport
• On Predicting Generalization using GANs
• The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs
• Explaining generalization in deep learning: progress and fundamental limits
• Gi and Pal Scores: Deep Neural Network Generalization Statistics