A Fourier View of REINFORCE
arXiv:1808.03953
Abstract
We show a connection between the Fourier spectrum of Boolean functions and the REINFORCE gradient estimator for binary latent variable models. We show that REINFORCE estimates (up to a factor) the degree-1 Fourier coefficients of a Boolean function. Using this connection we offer a new perspective on variance reduction in gradient estimation for latent variable models: namely, that variance reduction involves eliminating or reducing Fourier coefficients that do not have degree 1. We then use this connection to develop low-variance unbiased gradient estimators for binary latent variable models such as sigmoid belief networks. The estimator is based upon properties of the noise operator from Boolean Fourier theory and involves a sample-dependent baseline added to the REINFORCE estimator in a way that keeps the estimator unbiased. The baseline can be plugged into existing gradient estimators for further variance reduction.
References in corpus (6)
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Variational inference for Monte Carlo objectives
- Techniques for Learning Binary Stochastic Feedforward Neural Networks
- MuProp: Unbiased Backpropagation for Stochastic Neural Networks