From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
arXiv:1804.10974
Abstract
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, one extending RAML with fine-grained credit assignment and the other improving Actor-Critic with a systematic entropy regularization. On two benchmark datasets, we show the proposed algorithms outperform RAML and Actor-Critic respectively, providing new alternatives to sequence prediction.
arXiv commentACL 2018
References in corpus (14)
- Sequence to Sequence Learning with Neural Networks
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Sequence Level Training with Recurrent Neural Networks
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- An Actor-Critic Algorithm for Sequence Prediction
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Equivalence Between Policy Gradients and Soft Q-Learning
- Sequence-to-Sequence Learning as Beam-Search Optimization
- Reward Augmented Maximum Likelihood for Neural Structured Prediction
- Learning to Decode for Future Success
- Minimum Risk Training for Neural Machine Translation
- Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML