Stochastic Backpropagation through Mixture Density Distributions

Abstract

The ability to backpropagate stochastic gradients through continuous latent distributions has been crucial to the emergence of variational autoencoders and stochastic gradient variational Bayes. The key ingredient is an unbiased and low-variance way of estimating gradients with respect to distribution parameters from gradients evaluated at distribution samples. The "reparameterization trick" provides a class of transforms yielding such estimators for many continuous distributions, including the Gaussian and other members of the location-scale family. However the trick does not readily extend to mixture density models, due to the difficulty of reparameterizing the discrete distribution over mixture weights. This report describes an alternative transform, applicable to any continuous multivariate distribution with a differentiable density function from which samples can be drawn, and uses it to derive an unbiased estimator for mixture density weight derivatives. Combined with the reparameterization trick applied to the individual mixture components, this estimator makes it straightforward to train variational autoencoders with mixture-distributed latent variables, or to perform stochastic variational inference with a mixture density variational posterior.

References in corpus (3)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep AutoRegressive Networks

Cited by in corpus (16)

• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Bayesian Recurrent Neural Networks
• Monte Carlo Gradient Estimation in Machine Learning
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Pathwise Derivatives Beyond the Reparameterization Trick
• Normalizing Flows on Riemannian Manifolds
• Discrete Variational Autoencoders
• Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
• Differentiable Particle Filtering without Modifying the Forward Pass
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• Variational Marginal Particle Filters
• Automatic Differentiation Variational Inference with Mixtures
• Pathwise Derivatives for Multivariate Distributions
• Generalized Transformation-based Gradient
• Seq2Seq Mimic Games: A Signaling Perspective