A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
arXiv:1910.06419
Abstract
Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used throughout machine and reinforcement learning; however, they are usually explained as simple mathematical tricks without providing any insight into their nature. We use a first principles approach to explain LR and RP, and show a connection between the two via the divergence theorem. The theory motivated us to derive optimal importance sampling schemes to reduce LR gradient variance. Our newly derived distributions have analytic probability densities and can be directly sampled from. The improvement for Gaussian target distributions was modest, but for other distributions such as a Beta distribution, our method could lead to arbitrarily large improvements, and was crucial to obtain competitive performance in evolution strategies experiments.
arXiv comment8 pages + 19 pages appendix. Preliminary work
References in corpus (19)
- Categorical Reparameterization with Gumbel-Softmax
- Trust Region Policy Optimization
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Recurrent World Models Facilitate Policy Evolution
- Gradient Estimation Using Stochastic Computation Graphs
- The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Implicit Reparameterization Gradients
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
- Monte Carlo Gradient Estimation in Machine Learning
- Understanding and correcting pathologies in the training of learned optimizers
- The Mirage of Action-Dependent Baselines in Reinforcement Learning
- MuProp: Unbiased Backpropagation for Stochastic Neural Networks
- Boosting Trust Region Policy Optimization by Normalizing Flows Policy
- Overdispersed Black-Box Variational Inference
- Leveraging exploration in off-policy algorithms via normalizing flows
- Total stochastic gradient algorithms and applications in reinforcement learning
- New Tricks for Estimating Gradients of Expectations