ACtuAL: Actor-Critic Under Adversarial Learning
arXiv:1711.04755
Abstract
Generative Adversarial Networks (GANs) are a powerful framework for deep generative modeling. Posed as a two-player minimax problem, GANs are typically trained end-to-end on real-valued data and can be used to train a generator of high-dimensional and realistic images. However, a major limitation of GANs is that training relies on passing gradients from the discriminator through the generator via back-propagation. This makes it fundamentally difficult to train GANs with discrete data, as generation in this case typically involves a non-differentiable function. These difficulties extend to the reinforcement learning setting when the action space is composed of discrete decisions. We address these issues by reframing the GAN framework so that the generator is no longer trained using gradients through the discriminator, but is instead trained using a learned critic in the actor-critic framework with a Temporal Difference (TD) objective. This is a natural fit for sequence modeling and we use it to achieve improvements on language modeling tasks over the standard Teacher-Forcing methods.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Continuous control with deep reinforcement learning
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Improved Training of Wasserstein GANs
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Sequence Level Training with Recurrent Neural Networks
- SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Adversarial Learning for Neural Dialogue Generation
- Adversarial Generation of Natural Language
- Connecting Generative Adversarial Networks and Actor-Critic Methods
- Recurrent Batch Normalization
- Adversarially Regularized Autoencoders
- MuProp: Unbiased Backpropagation for Stochastic Neural Networks
- Learning Simple Algorithms from Examples