Estimating Model Uncertainty of Neural Networks in Sparse Information Form

Abstract

We present a sparse representation of model uncertainty for Deep Neural Networks (DNNs) where the parameter posterior is approximated with an inverse formulation of the Multivariate Normal Distribution (MND), also known as the information form. The key insight of our work is that the information matrix, i.e. the inverse of the covariance matrix tends to be sparse in its spectrum. Therefore, dimensionality reduction techniques such as low rank approximations (LRA) can be effectively exploited. To achieve this, we develop a novel sparsification algorithm and derive a cost-effective analytical sampler. As a result, we show that the information form can be scalably applied to represent model uncertainty in DNNs. Our exhaustive theoretical analysis and empirical evaluations on various benchmarks show the competitiveness of our approach over the current methods.

arXiv commentAccepted to the Thirty-seventh International Conference on Machine Learning (ICML) 2020

References in corpus (13)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Overcoming catastrophic forgetting in neural networks
• Weight Uncertainty in Neural Networks
• Continual Learning Through Synaptic Intelligence
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• 'In-Between' Uncertainty in Bayesian Neural Networks
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Variational Laplace Autoencoders
• Fast symmetric factorization of hierarchical matrices with applications

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Explaining Bayesian Neural Networks
• Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry
• Bayesian Optimization Meets Laplace Approximation for Robotic Introspection