Copula-like Variational Inference

Abstract

This paper considers a new family of variational distributions motivated by Sklar's theorem. This family is based on new copula-like densities on the hypercube with non-uniform marginals which can be sampled efficiently, i.e. with a complexity linear in the dimension of state space. Then, the proposed variational densities that we suggest can be seen as arising from these copula-like densities used as base distributions on the hypercube with Gaussian quantile functions and sparse rotation matrices as normalizing flows. The latter correspond to a rotation of the marginals with complexity $\mathcal{O}(d\log d)$. We provide some empirical evidence that such a variational family can also approximate non-Gaussian posteriors and can be beneficial compared to Gaussian approximations. Our method performs largely comparably to state-of-the-art variational approximations on standard regression and classification benchmarks for Bayesian Neural Networks.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada

References in corpus (25)

• Auto-Encoding Variational Bayes
• Variational Inference with Normalizing Flows
• Density estimation using Real NVP
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Bayesian Compression for Deep Learning
• MADE: Masked Autoencoder for Distribution Estimation
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Implicit Reparameterization Gradients
• Sylvester Normalizing Flows for Variational Inference
• Hyperspherical Variational Auto-Encoders
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Improving Variational Auto-Encoders using Householder Flow
• Variational Inference using Implicit Distributions
• Variational Boosting: Iteratively Refining Posterior Approximations
• Boosting Variational Inference
• Nonparametric variational inference
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Variational Gaussian Copula Inference
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Unbiased Implicit Variational Inference
• Fast Approximation of Rotations and Hessians matrices
• Boosting Variational Inference: an Optimization Perspective
• Boosting Black Box Variational Inference
• On approximating copulas by finite mixtures

Cited by in corpus (1)

• Variational Marginal Particle Filters