NeurIPS 2020 Competition: Predicting Generalization in Deep Learning

Abstract

Understanding generalization in deep learning is arguably one of the most important questions in deep learning. Deep learning has been successfully adopted to a large number of problems ranging from pattern recognition to complex decision making, but many recent researchers have raised many concerns about deep learning, among which the most important is generalization. Despite numerous attempts, conventional statistical learning approaches have yet been able to provide a satisfactory explanation on why deep learning works. A recent line of works aims to address the problem by trying to predict the generalization performance through complexity measures. In this competition, we invite the community to propose complexity measures that can accurately predict generalization of models. A robust and general complexity measure would potentially lead to a better understanding of deep learning's underlying mechanism and behavior of deep models on unseen data, or shed light on better generalization bounds. All these outcomes will be important for making deep learning more robust and reliable.

20 pages, 2 figures. Accepted for NeurIPS 2020 Competitions Track. Lead organizer: Yiding Jiang

References in corpus (13)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Learning both Weights and Connections for Efficient Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Self-Distillation Amplifies Regularization in Hilbert Space
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Learning
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Towards Task and Architecture-Independent Generalization Gap Predictors
• Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes

Cited by in corpus (10)

• Assessing Generalization of SGD via Disagreement
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Evaluation of Complexity Measures for Deep Learning Generalization in Medical Image Analysis
• Measuring Generalization with Optimal Transport
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
• On Predicting Generalization using GANs
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Explaining generalization in deep learning: progress and fundamental limits
• Auditing Robot Learning for Safety and Compliance during Deployment