A Bayesian Perspective on Training Speed and Model Selection

Abstract

We take a Bayesian perspective to illustrate a connection between training speed and the marginal likelihood in linear models. This provides two major insights: first, that a measure of a model's training speed can be used to estimate its marginal likelihood. Second, that this measure, under certain conditions, predicts the relative weighting of models in linear model combinations trained to minimize a regression loss. We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks. We further provide encouraging empirical evidence that the intuition developed in these settings also holds for deep neural networks trained with stochastic gradient descent. Our results suggest a promising new direction towards explaining why neural networks trained with stochastic gradient descent are biased towards functions that generalize well.

To be presented at NeurIPS 2020

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Train faster, generalize better: Stability of stochastic gradient descent
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Uniform convergence may be unable to explain generalization in deep learning
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Stiffness: A New Perspective on Generalization in Neural Networks
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Learning Invariances using the Marginal Likelihood
• Bayesian Deep Ensembles via the Neural Tangent Kernel

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Repulsive Deep Ensembles are Bayesian
• A User-Guided Bayesian Framework for Ensemble Feature Selection in Life Science Applications (UBayFS)
• Deep Active Learning by Leveraging Training Dynamics
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding