Connecting the Dots Between MLE and RL for Sequence Prediction

Abstract

Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently distinct algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of a reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, inspired from the framework, we present a new algorithm that dynamically interpolates among the family of algorithms for scheduled sequence model learning. Experiments on machine translation, text summarization, and game imitation learning demonstrate the superiority of the proposed algorithm.

arXiv commentMajor revision. The first two authors contributed equally

References in corpus (18)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Trust Region Policy Optimization
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Spike and slab variable selection: Frequentist and Bayesian strategies
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• MaskGAN: Better Text Generation via Filling in the______
• Selective Encoding for Abstractive Sentence Summarization
• Maximum a Posteriori Policy Optimisation
• Equivalence Between Policy Gradients and Soft Q-Learning
• Data Noising as Smoothing in Neural Network Language Models
• Unsupervised Text Style Transfer using Language Models as Discriminators
• Cold-Start Reinforcement Learning with Softmax Policy Gradient
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
• Policy Shaping and Generalized Update Equations for Semantic Parsing from Denotations

Cited by in corpus (5)

• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Nested-Wasserstein Self-Imitation Learning for Sequence Generation
• ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models