On Power Laws in Deep Ensembles
arXiv:2007.08483
Abstract
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classification problem and investigate the behavior of both non-calibrated and calibrated negative log-likelihood (CNLL) of a deep ensemble as a function of the ensemble size and the member network size. We indicate the conditions under which CNLL follows a power law w.r.t. ensemble size or member network size, and analyze the dynamics of the parameters of the discovered power laws. Our important practical finding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters (we call this ensemble a memory split). Using the detected power law-like dependencies, we can predict (1) the possible gain from the ensembling of networks with given structure, (2) the optimal memory split given a memory budget, based on a relatively small number of trained networks. We describe the memory split advantage effect in more details in arXiv:2005.07292
Published in NeurIPS 2020 and Workshop on Uncertainty and Robustness in Deep Learning at ICML 2020
References in corpus (19)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Reconciling modern machine learning practice and the bias-variance trade-off
- Scaling Laws for Neural Language Models
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- Deep Ensembles: A Loss Landscape Perspective
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Bayesian Deep Learning and a Probabilistic Perspective of Generalization
- Scaling description of generalization with number of parameters in deep learning
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- A Modern Take on the Bias-Variance Tradeoff in Neural Networks
- Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
- Bayesian Deep Ensembles via the Neural Tangent Kernel
- When Ensembling Smaller Models is More Efficient than Single Large Models
- EnsembleNet: End-to-End Optimization of Multi-headed Models
- Diversity inducing Information Bottleneck in Model Ensembles
- Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?