A Study of Reinforcement Learning for Neural Machine Translation

Abstract

Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation (NMT) system. However, due to its instability, successfully RL training is challenging, especially in real-world systems where deep models and large datasets are leveraged. In this paper, taking several large-scale translation tasks as testbeds, we conduct a systematic study on how to train better NMT models using reinforcement learning. We provide a comprehensive comparison of several important factors (e.g., baseline reward, reward shaping) in RL training. Furthermore, to fill in the gap that it remains unclear whether RL is still beneficial when monolingual data is used, we propose a new method to leverage RL to further boost the performance of NMT systems trained with source/target monolingual data. By integrating all our findings, we obtain competitive results on WMT14 English- German, WMT17 English-Chinese, and WMT17 Chinese-English translation tasks, especially setting a state-of-the-art performance on WMT17 Chinese-English translation task.

arXiv commentEMNLP 2018

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• Achieving Human Parity on Automatic Chinese to English News Translation
• An Actor-Critic Algorithm for Sequence Prediction
• Adversarial Learning for Neural Dialogue Generation
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Adversarial Neural Machine Translation
• The University of Edinburgh's Neural MT Systems for WMT17
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Cited by in corpus (13)

• Generating Persona Consistent Dialogues by Exploiting Natural Language Inference
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• A Reinforced Generation of Adversarial Examples for Neural Machine Translation
• Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning
• Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
• Microwave Integrated Circuits Design with Relational Induction Neural Network
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs
• Posterior-GAN: Towards Informative and Coherent Response Generation with Posterior Generative Adversarial Network
• A Study of State Aliasing in Structured Prediction with RNNs
• Optimizing AD Pruning of Sponsored Search with Reinforcement Learning