Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
arXiv:2104.04975
Abstract
Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In this work, we present a scalable marginal-likelihood estimation method to select both hyperparameters and network architectures, based on the training data alone. Some hyperparameters can be estimated online during training, simplifying the procedure. Our marginal-likelihood estimate is based on Laplace's method and Gauss-Newton approximations to the Hessian, and it outperforms cross-validation and manual-tuning on standard regression and image classification datasets, especially in terms of calibration and out-of-distribution detection. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable (e.g., in nonstationary settings).
arXiv commentICML 2021
References in corpus (10)
- On Calibration of Modern Neural Networks
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Practical Deep Learning with Bayesian Principles
- Marginal likelihood computation for model selection and hypothesis testing: an extensive review
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Learning Invariances using the Marginal Likelihood
- Practical Gauss-Newton Optimisation for Deep Learning
- 'In-Between' Uncertainty in Bayesian Neural Networks