An Actor-Critic Algorithm for Sequence Prediction

Abstract

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a \textit{critic} network that is trained to predict the value of an output token, given the policy of an \textit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

References in corpus (5)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Blocks and Fuel: Frameworks for deep learning

Cited by in corpus (15)

• Professor Forcing: A New Algorithm for Training Recurrent Networks
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Learning to Decode for Future Success
• Neural Text Generation: A Practical Guide
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• Transcribing Content from Structural Images with Spotlight Mechanism
• Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
• Comprehension-guided referring expressions
• Reinforcement Learning for Learning Rate Control
• Trainable Greedy Decoding for Neural Machine Translation
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Improving End-to-End Speech Recognition with Policy Learning
• Modeling Past and Future for Neural Machine Translation
• Neural Sequence Model Training via $\alpha$-divergence Minimization