Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

Abstract

We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior in a variational setting for Bayesian neural networks. We show that through this interpretation it is both efficient and straightforward to improve the approximation by employing normalizing flows while still allowing for local reparametrizations and a tractable lower bound. In experiments we show that with this new approximation we can significantly improve upon classical mean field for Bayesian neural networks on both predictive accuracy as well as predictive uncertainty.

Appearing at the International Conference on Machine Learning (ICML) 2017

References in corpus (7)

• Explaining and Harnessing Adversarial Examples
• Expectation Propagation for approximate Bayesian inference
• Understanding deep learning requires rethinking generalization
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

Cited by in corpus (108)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• An Introduction to Variational Autoencoders
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Deep Ensembles: A Loss Landscape Perspective
• Normalizing Flows for Probabilistic Modeling and Inference
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• Lifelong Generative Modeling
• What Are Bayesian Neural Network Posteriors Really Like?
• Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities
• Hypernetwork functional image representation
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Quantum Hamiltonian-Based Models and the Variational Quantum Thermalizer Algorithm
• Ensemble Neural Networks (ENN): A gradient-free stochastic method
• Do Deep Generative Models Know What They Don't Know?
• Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Study
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Calibration of Encoder Decoder Models for Neural Machine Translation
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Adversarial Distributional Training for Robust Deep Learning
• The Functional Neural Process
• Stabilizing Invertible Neural Networks Using Mixture Models
• Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• Neural Spline Flows
• Are Generative Classifiers More Robust to Adversarial Attacks?
• Uncertainty-Aware Reliable Text Classification
• Semi-Implicit Variational Inference
• Bayesian Neural Network Priors Revisited
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Recent advances in deep learning theory
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Transferable Calibration with Lower Bias and Variance in Domain Adaptation
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Confidence-Aware Learning for Camouflaged Object Detection
• Uncertainty quantification of molecular property prediction with Bayesian neural networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Randomized Value Functions via Multiplicative Normalizing Flows
• Quantifying Classification Uncertainty using Regularized Evidential Neural Networks
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Calibration of Model Uncertainty for Dropout Variational Inference
• Predictive Uncertainty Quantification with Compound Density Networks
• Being Bayesian about Categorical Probability
• Testing the robustness of simulation-based gravitational-wave population inference
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles
• DeepLens: Interactive Out-of-distribution Data Detection in NLP Models
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Estimating Photometric Redshifts for Galaxies from the DESI Legacy Imaging Surveys with Bayesian Neural Networks Trained by DESI EDR
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
• Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
• Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
• Stochastic Neural Network with Kronecker Flow
• OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• Variational Inference with Continuously-Indexed Normalizing Flows
• Distributed Weight Consolidation: A Brain Segmentation Case Study
• Posterior Meta-Replay for Continual Learning
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Bayesian posterior approximation with stochastic ensembles
• Uncertainty Quantification and Software Risk Analysis for Digital Twins in the Nearly Autonomous Management and Control Systems: A Review
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• A Bayesian neural network predicts the dissolution of compact planetary systems
• Variational Bayes: A report on approaches and applications
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Knowing what you know in brain segmentation using Bayesian deep neural networks
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Robustness via Cross-Domain Ensembles
• Are Bayesian neural networks intrinsically good at out-of-distribution detection?
• Uncertainty quantification of molecular property prediction using Bayesian neural network models
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Copula-like Variational Inference
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation
• Scalable Bayesian neural networks by layer-wise input augmentation
• Training Invertible Linear Layers through Rank-One Perturbations
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• Uncertainty Quantification in Deep Residual Neural Networks
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification
• Modeling Graph Node Correlations with Neighbor Mixture Models
• Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases
• Decision Theoretic Bootstrapping
• Leveraging Uncertainty from Deep Learning for Trustworthy Materials Discovery Workflows
• Pathologies in priors and inference for Bayesian transformers
• Energy-based Unknown Intent Detection with Data Manipulation
• Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation
• Uncertainty estimation under model misspecification in neural network regression
• Frustratingly Easy Uncertainty Estimation for Distribution Shift
• Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference
• Transformation Models for Flexible Posteriors in Variational Bayes