Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
arXiv:1807.09511
Abstract
In real-world scenarios, it is appealing to learn a model carrying out stochastic operations internally, known as stochastic computation graphs (SCGs), rather than learning a deterministic mapping. However, standard backpropagation is not applicable to SCGs. We attempt to address this issue from the angle of cost propagation, with local surrogate costs, called Q-functions, constructed and learned for each stochastic node in an SCG. Then, the SCG can be trained based on these surrogate costs using standard backpropagation. We propose the entire framework as a solution to generalize backpropagation for SCGs, which resembles an actor-critic architecture but based on a graph. For broad applicability, we study a variety of SCG structures from one cost to multiple costs. We utilize recent advances in reinforcement learning (RL) and variational Bayes (VB), such as off-policy critic learning and unbiased-and-low-variance gradient estimation, and review them in the context of SCGs. The generalized backpropagation extends transported learning signals beyond gradients between stochastic nodes while preserving the benefit of backpropagating gradients through deterministic nodes. Experimental suggestions and concerns are listed to help design and test any specific model using this framework.
arXiv commentNeurIPS 2018 Deep Reinforcement Learning Workshop
References in corpus (18)
- Continuous control with deep reinforcement learning
- Categorical Reparameterization with Gumbel-Softmax
- Trust Region Policy Optimization
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Sequence Level Training with Recurrent Neural Networks
- Towards Biologically Plausible Deep Learning
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Gradient Estimation Using Stochastic Computation Graphs
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Variational Bayesian Inference with Stochastic Search
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
- Decoupled Neural Interfaces using Synthetic Gradients
- MuProp: Unbiased Backpropagation for Stochastic Neural Networks
- Local Expectation Gradients for Doubly Stochastic Variational Inference