Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

Abstract

Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.

Published at ICLR 2018

References in corpus (5)

• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

Cited by in corpus (48)

• Fast Decoding in Sequence Models using Discrete Latent Variables
• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• Pathwise Derivatives Beyond the Reparameterization Trick
• Theory and Experiments on Vector Quantized Autoencoders
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• A Tutorial on Deep Latent Variable Models of Natural Language
• Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Adversarial Contrastive Estimation
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures
• Reparameterization Gradient for Non-differentiable Models
• Using Large Ensembles of Control Variates for Variational Inference
• Solving Quantum Statistical Mechanics with Variational Autoregressive Networks and Quantum Circuits
• Straight-Through Estimator as Projected Wasserstein Gradient Flow
• Improved Gradient-Based Optimization Over Discrete Distributions
• GO Gradient for Expectation-Based Objectives
• Beyond variance reduction: Understanding the true impact of baselines on policy optimization
• Improving Evolutionary Strategies with Generative Neural Networks
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Variational Rejection Sampling
• Reward-estimation variance elimination in sequential decision processes
• Foveation for Segmentation of Ultra-High Resolution Images
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Semi-supervised Sequential Generative Models
• Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Pathwise Derivatives for Multivariate Distributions
• Programming with Neural Surrogates of Programs
• ReGAN: RE[LAX|BAR|INFORCE] based Sequence Generation using GANs
• Neural Conditional Event Time Models
• Imitation Learning of Factored Multi-agent Reactive Models
• Hierarchical Variational Imitation Learning of Control Programs
• Discrete Action On-Policy Learning with Action-Value Critic
• Cooperative image captioning
• Learning Generalized Gumbel-max Causal Mechanisms
• KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Joint Mapping and Calibration via Differentiable Sensor Fusion
• Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• Deep Learning for Wireless Communications
• Latent Transformations for Discrete-Data Normalising Flows
• GO Hessian for Expectation-Based Objectives
• Lattice Representation Learning
• Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy
• ARSM Gradient Estimator for Supervised Learning to Rank