An Actor-Critic Algorithm for Sequence Prediction

Abstract

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a \textit{critic} network that is trained to predict the value of an output token, given the policy of an \textit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

References in corpus (11)

• Continuous control with deep reinforcement learning
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Sequence Level Training with Recurrent Neural Networks
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Blocks and Fuel: Frameworks for deep learning
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Minimum Risk Training for Neural Machine Translation
• Learning Simple Algorithms from Examples

Cited by in corpus (114)

• Deep Reinforcement Learning: An Overview
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Adversarial Learning for Neural Dialogue Generation
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Clinically Accurate Chest X-Ray Report Generation
• Sequence-to-Sequence Learning as Beam-Search Optimization
• A review on deep reinforcement learning for fluid mechanics: an update
• A review on Deep Reinforcement Learning for Fluid Mechanics
• CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Adversarial Neural Machine Translation
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Recursively Summarizing Books with Human Feedback
• Fast Structured Decoding for Sequence Models
• DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
• Learning to Decode for Future Success
• Multi-layer Representation Fusion for Neural Machine Translation
• Towards Binary-Valued Gates for Robust LSTM Training
• Learning to summarize from human feedback
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Neural Text Generation: A Practical Guide
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Transcribing Content from Structural Images with Spotlight Mechanism
• Learning to Segment Inputs for NMT Favors Character-Level Processing
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Video Captioning via Hierarchical Reinforcement Learning
• A Survey on Neural Network-Based Summarization Methods
• Paraphrase Generation with Deep Reinforcement Learning
• A Study of Reinforcement Learning for Neural Machine Translation
• Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
• Comprehension-guided referring expressions
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Explaining and Generalizing Back-Translation through Wake-Sleep
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Reinforcement Learning for Learning Rate Control
• Trainable Greedy Decoding for Neural Machine Translation
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Ad Headline Generation using Self-Critical Masked Language Model
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Multimodal Machine Translation with Reinforcement Learning
• A Reinforced Generation of Adversarial Examples for Neural Machine Translation
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• A New Approach for Resource Scheduling with Deep Reinforcement Learning
• Action Assembly: Sparse Imitation Learning for Text Based Games with Combinatorial Action Spaces
• Hint-Based Training for Non-Autoregressive Machine Translation
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
• Unpaired Image Captioning by Language Pivoting
• Neural Phrase-to-Phrase Machine Translation
• Double Path Networks for Sequence to Sequence Learning
• Efficient Collaborative Multi-Agent Deep Reinforcement Learning for Large-Scale Fleet Management
• Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• Improving End-to-End Speech Recognition with Policy Learning
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
• Efficiency Metrics for Data-Driven Models: A Text Summarization Case Study
• Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation
• Distributional Reinforcement Learning for Energy-Based Sequential Models
• Rethinking Exposure Bias In Language Modeling
• Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approach
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Sequence Generation with Guider Network
• Optimal Completion Distillation for Sequence Learning
• Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks
• Closed-Book Training to Improve Summarization Encoder Memory
• PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policies
• Sparse Stochastic Zeroth-Order Optimization with an Application to Bandit Structured Prediction
• Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• BanditRank: Learning to Rank Using Contextual Bandits
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Representation Learning of Pedestrian Trajectories Using Actor-Critic Sequence-to-Sequence Autoencoder
• Loss Bounds for Approximate Influence-Based Abstraction
• Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs
• XL-NBT: A Cross-lingual Neural Belief Tracking Framework
• Goal-directed Generation of Discrete Structures with Conditional Generative Models
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Towards Coherent and Cohesive Long-form Text Generation
• Promising Accurate Prefix Boosting for sequence-to-sequence ASR
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation
• Sample Efficient Ensemble Learning with Catalyst.RL
• Learning How to Self-Learn: Enhancing Self-Training Using Neural Reinforcement Learning
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Modeling Past and Future for Neural Machine Translation
• Towards one-shot learning for rare-word translation with external experts
• Multi-Agent Decentralized Belief Propagation on Graphs
• Transfer Reward Learning for Policy Gradient-Based Text Generation
• Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System
• Deep Reinforcement Learning for Personalized Search Story Recommendation
• Analysing Dropout and Compounding Errors in Neural Language Models
• Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation
• Optimizing AD Pruning of Sponsored Search with Reinforcement Learning
• Generating summaries tailored to target characteristics
• A Reinforcement Learning Approach for Sequential Spatial Transformer Networks
• Autoregressive Knowledge Distillation through Imitation Learning
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning
• Imitation Learning for Neural Morphological String Transduction
• Approximate Distribution Matching for Sequence-to-Sequence Learning
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• Neural Sequence Model Training via $\alpha$-divergence Minimization
• Improving Adversarial Text Generation by Modeling the Distant Future
• Adversarial Machine Learning in Text Analysis and Generation
• A Smart Sliding Chinese Pinyin Input Method Editor on Touchscreen