Implicit Weight Uncertainty in Neural Networks

Abstract

Modern neural networks tend to be overconfident on unseen, noisy or incorrectly labelled data and do not produce meaningful uncertainty measures. Bayesian deep learning aims to address this shortcoming with variational approximations (such as Bayes by Backprop or Multiplicative Normalising Flows). However, current approaches have limitations regarding flexibility and scalability. We introduce Bayes by Hypernet (BbH), a new method of variational approximation that interprets hypernetworks as implicit distributions. It naturally uses neural networks to model arbitrarily complex distributions and scales to modern deep learning architectures. In our experiments, we demonstrate that our method achieves competitive accuracies and predictive uncertainties on MNIST and a CIFAR5 task, while being the most robust against adversarial attacks.

Submitted to NIPS 2018, under review

References in corpus (6)

• On Calibration of Modern Neural Networks
• Expectation Propagation for approximate Bayesian inference
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Variational Inference using Implicit Distributions

Cited by in corpus (27)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy
• Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation
• Principled Weight Initialization for Hypernetworks
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Adversarial Attack for Uncertainty Estimation: Identifying Critical Regions in Neural Networks
• Predictive Uncertainty Quantification with Compound Density Networks
• Hypermodels for Exploration
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• VFunc: a Deep Generative Model for Functions
• Stochastic Neural Network with Kronecker Flow
• Posterior Meta-Replay for Continual Learning
• Uncertainty Quantification and Software Risk Analysis for Digital Twins in the Nearly Autonomous Management and Control Systems: A Review
• Graceful Degradation and Related Fields
• Uncertainty-Aware Deep Classifiers using Generative Models
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Are Bayesian neural networks intrinsically good at out-of-distribution detection?
• MetaInv-Net: Meta Inversion Network for Sparse View CT Image Reconstruction
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• StelNet: Hierarchical Neural Network for Automatic Inference in Stellar Characterization
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Uncertainty in Neural Relational Inference Trajectory Reconstruction