Stationary Activations for Uncertainty Calibration in Deep Learning

Abstract

We introduce a new family of non-linear neural network activation functions that mimic the properties induced by the widely-used Matérn family of kernels in Gaussian process (GP) models. This class spans a range of locally stationary models of various degrees of mean-square differentiability. We show an explicit link to the corresponding GP models in the case that the network consists of one infinitely wide hidden layer. In the limit of infinite smoothness the Matérn family results in the RBF kernel, and in this case we recover RBF activations. Matérn activation functions result in similar appealing properties to their counterparts in GP models, and we demonstrate that the local stationarity property together with limited mean-square differentiability shows both good performance and uncertainty calibration in Bayesian deep learning tasks. In particular, local stationarity helps calibrate out-of-distribution (OOD) uncertainty. We demonstrate these properties on classification and regression benchmarks and a radar emitter classification task.

arXiv commentTo appear in Advances in Neural Information Processing Systems (NeurIPS 2020)

References in corpus (15)

• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Neural Networks as Gaussian Processes
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Functional Variational Bayesian Neural Networks
• Variational Fourier features for Gaussian processes
• Reducing Network Agnostophobia
• Do Deep Generative Models Know What They Don't Know?
• Noise Contrastive Priors for Functional Uncertainty
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• A randomized gradient-free attack on ReLU networks

Cited by in corpus (2)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Few-shot time series segmentation using prototype-defined infinite hidden Markov models