Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty

Abstract

From the statistical learning perspective, complexity control via explicit regularization is a necessity for improving the generalization of over-parameterized models. However, the impressive generalization performance of neural networks with only implicit regularization may be at odds with this conventional wisdom. In this work, we revisit the importance of explicit regularization for obtaining well-calibrated predictive uncertainty. Specifically, we introduce a probabilistic measure of calibration performance, which is lower bounded by the log-likelihood. We then explore explicit regularization techniques for improving the log-likelihood on unseen samples, which provides well-calibrated predictive uncertainty. Our findings present a new direction to improve the predictive probability quality of deterministic neural networks, which can be an efficient and scalable alternative to Bayesian neural networks and ensemble methods.

References in corpus (15)

• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• HDLTex: Hierarchical Deep Learning for Text Classification
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Practical Deep Learning with Bayesian Principles
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Uncertainty in the Variational Information Bottleneck
• Towards Understanding Regularization in Batch Normalization
• Hybrid Models with Deep and Invertible Features
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric