REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

Abstract

Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work (Jang et al. 2016, Maddison et al. 2016) has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, \emph{unbiased} gradient estimates. Then, we introduce a modification to the continuous relaxation and show that the tightness of the relaxation can be adapted online, removing it as a hyperparameter. We show state-of-the-art variance reduction on several benchmark generative modeling tasks, generally leading to faster convergence to a better final log-likelihood.

arXiv commentNIPS 2017

Cited by in corpus (99)

• NVAE: A Deep Hierarchical Variational Autoencoder
• Learning Discrete Structures for Graph Neural Networks
• Fast Decoding in Sequence Models using Discrete Latent Variables
• From Variational to Deterministic Autoencoders
• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• Monte Carlo Gradient Estimation in Machine Learning
• Pathwise Derivatives Beyond the Reparameterization Trick
• A Factor Graph Approach to Automated Design of Bayesian Signal Processing Algorithms
• Robust Training of Vector Quantized Bottleneck Models
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• A Tutorial on Deep Latent Variable Models of Natural Language
• Learning Linear Dynamical Systems with Semi-Parametric Least Squares
• Advances in Variational Inference
• Gradient Estimation with Stochastic Softmax Tricks
• Intensity-Free Learning of Temporal Point Processes
• A RAD approach to deep mixture models
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Neural Control Variates for Variance Reduction
• Estimating Gradients for Discrete Random Variables by Sampling without Replacement
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• Vector representations of text data in deep learning
• Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
• DiCE: The Infinitely Differentiable Monte-Carlo Estimator
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• DisARM: An Antithetic Gradient Estimator for Binary Latent Variables
• Reparameterization Gradient for Non-differentiable Models
• Using Large Ensembles of Control Variates for Variational Inference
• Radial and Directional Posteriors for Bayesian Neural Networks
• Storchastic: A Framework for General Stochastic Automatic Differentiation
• Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design
• Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
• Sparse Graph Attention Networks
• Solving Quantum Statistical Mechanics with Variational Autoregressive Networks and Quantum Circuits
• Differentiable lower bound for expected BLEU score
• Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions
• Amortized Bethe Free Energy Minimization for Learning MRFs
• UFO-BLO: Unbiased First-Order Bilevel Optimization
• CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generation
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Straight-Through Estimator as Projected Wasserstein Gradient Flow
• Improved Gradient-Based Optimization Over Discrete Distributions
• Generalizable Adversarial Attacks with Latent Variable Perturbation Modelling
• GO Gradient for Expectation-Based Objectives
• On Machine Learning and Structure for Mobile Robots
• Learning to learn generative programs with Memoised Wake-Sleep
• Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems
• The Thermodynamic Variational Objective
• Generative Semantic Hashing Enhanced via Boltzmann Machines
• Cooperative Learning of Disjoint Syntax and Semantics
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models
• Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning
• Coupled Gradient Estimators for Discrete Latent Variables
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• Variance Reduction for Evolution Strategies via Structured Control Variates
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Variational Rejection Sampling
• Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation
• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator
• Neural Image Compression and Explanation
• New Tricks for Estimating Gradients of Expectations
• Semi-supervised Sequential Generative Models
• Foveation for Segmentation of Ultra-High Resolution Images
• MIM: Mutual Information Machine
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds
• Double Control Variates for Gradient Estimation in Discrete Latent Variable Models
• CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator
• Discrete Action On-Policy Learning with Action-Value Critic
• Imitation Learning of Factored Multi-agent Reactive Models
• ${\ell }_0$-based Sparse Canonical Correlation Analysis
• A Rule for Gradient Estimator Selection, with an Application to Variational Inference
• Neural Conditional Event Time Models
• Learning with Multiplicative Perturbations
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Pathwise Derivatives for Multivariate Distributions
• Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows
• Cooperative image captioning
• Fitting summary statistics of neural data with a differentiable spiking network simulator
• ARSM Gradient Estimator for Supervised Learning to Rank
• Using stochastic computation graphs formalism for optimization of sequence-to-sequence model
• Learning Sparsity of Representations with Discrete Latent Variables
• Debiasing a First-order Heuristic for Approximate Bi-level Optimization
• Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces
• Multilevel Monte Carlo Variational Inference
• Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy
• Hindsight Network Credit Assignment
• Modeling Graph Node Correlations with Neighbor Mixture Models
• Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface
• Differentiable Greedy Submodular Maximization: Guarantees, Gradient Estimators, and Applications
• A Fourier View of REINFORCE
• Lattice Representation Learning
• Joint Mapping and Calibration via Differentiable Sensor Fusion
• Approximation Based Variance Reduction for Reparameterization Gradients