Noise Contrastive Priors for Functional Uncertainty

Abstract

Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of an independent normal prior in weight space imposes relatively weak constraints on the function posterior, allowing it to generalize in unforeseen ways on inputs outside of the training distribution. We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates. The key idea is to train the model to output high uncertainty for data points outside of the training distribution. NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs. NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training. Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning. We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.

arXiv comment12 pages, 6 figures

References in corpus (3)

• Sequence to Sequence Learning with Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Bayesian Compression for Deep Learning

Cited by in corpus (28)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• The Case for Bayesian Deep Learning
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• AI Research Considerations for Human Existential Safety (ARCHES)
• The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
• Bayesian Neural Network Priors Revisited
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• STUaNet: Understanding uncertainty in spatiotemporal collective human mobility
• Deep Evidential Regression
• Diversity inducing Information Bottleneck in Model Ensembles
• Output-Constrained Bayesian Neural Networks
• Safe Interactive Model-Based Learning
• Being Bayesian about Categorical Probability
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Incorporating Interpretable Output Constraints in Bayesian Neural Networks
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
• Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration
• Graceful Degradation and Related Fields
• Bigeminal Priors Variational auto-encoder
• Robustness via Cross-Domain Ensembles
• Functional Space Variational Inference for Uncertainty Estimation in Computer Aided Diagnosis
• Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Multivariate Deep Evidential Regression
• Active Hierarchical Imitation and Reinforcement Learning