Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

Abstract

We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian \cite{gupta1999matrix} parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" \cite{kingma2015variational} on this posterior distribution we arrive at a Gaussian Process \cite{rasmussen2006gaussian} interpretation of the hidden units in each layer and we, similarly with \cite{gal2015dropout}, provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" \cite{snelson2005sparse} in our model, which in turn allows for more efficient sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.

Updated results with the original folds in the regression experiments. Appearing in the International Conference on Machine Learning (ICML) 2016

References in corpus (5)

• Expectation Propagation for approximate Bayesian inference
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring

Cited by in corpus (55)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• An Introduction to Variational Autoencoders
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Lifelong Generative Modeling
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Control Barriers in Bayesian Learning of System Dynamics
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Tighter risk certificates for neural networks
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Adversarial Distributional Training for Robust Deep Learning
• The Functional Neural Process
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• Probabilistic Safety Constraints for Learned High Relative Degree System Dynamics
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Action and Perception as Divergence Minimization
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• DeepGLEAM: A hybrid mechanistic and deep learning model for COVID-19 forecasting
• Safe Multi-Agent Interaction through Robust Control Barrier Functions with Learned Uncertainties
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• Deep Gaussian Process-Based Bayesian Inference for Contaminant Source Localization
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Bayesian Deep Learning via Subnetwork Inference
• Predictive Uncertainty Quantification with Compound Density Networks
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
• Radial and Directional Posteriors for Bayesian Neural Networks
• W2WNet: a two-module probabilistic Convolutional Neural Network with embedded data cleansing functionality
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• The Bayesian Learning Rule
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Tractable Approximate Gaussian Inference for Bayesian Neural Networks
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Uncertainty-aware Safe Exploratory Planning using Gaussian Process and Neural Control Contraction Metric
• Variational Bayes: A report on approaches and applications
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Robustness via Cross-Domain Ensembles
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• Copula-like Variational Inference
• An Interpretable and Uncertainty Aware Multi-Task Framework for Multi-Aspect Sentiment Analysis
• Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification
• Efficient Approximate Inference with Walsh-Hadamard Variational Inference
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Pathologies in priors and inference for Bayesian transformers
• Stochastic Bayesian Neural Networks
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift