Reward Augmented Maximum Likelihood for Neural Structured Prediction

Abstract

A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation. This paper presents a simple and computationally efficient approach to incorporate task reward into a maximum likelihood framework. By establishing a link between the log-likelihood and expected reward objectives, we show that an optimal regularized expected reward is achieved when the conditional distribution of the outputs given the inputs is proportional to their exponentiated scaled rewards. Accordingly, we present a framework to smooth the predictive probability of the outputs using their corresponding rewards. We optimize the conditional log-probability of augmented outputs that are sampled proportionally to their exponentiated scaled rewards. Experiments on neural sequence to sequence models for speech recognition and machine translation show notable improvements over a maximum likelihood baseline by using reward augmented maximum likelihood (RAML), where the rewards are defined as the negative edit distance between the outputs and the ground truth labels.

NIPS 2016

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Listen, Attend and Spell
• Optimal control as a graphical model inference problem
• An Actor-Critic Algorithm for Sequence Prediction
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Task Loss Estimation for Sequence Prediction
• Loss-sensitive Training of Probabilistic Conditional Random Fields

Cited by in corpus (31)

• A Deep Reinforced Model for Abstractive Summarization
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Combining policy gradient and Q-learning
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Controlling Information Capacity of Binary Neural Network
• Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education
• A Divergence Minimization Perspective on Imitation Learning Methods
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Improving Regression Performance with Distributional Losses
• SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
• Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence
• Optimal Completion Distillation for Sequence Learning
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding
• Closed-Book Training to Improve Summarization Encoder Memory
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Bottom-Up Meta-Policy Search
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Improving Text Generation with Student-Forcing Optimal Transport
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Sample Efficient Learning of Image-Based Diagnostic Classifiers Using Probabilistic Labels
• Analysing Dropout and Compounding Errors in Neural Language Models
• Approximate Distribution Matching for Sequence-to-Sequence Learning
• Investigating the Decoders of Maximum Likelihood Sequence Models: A Look-ahead Approach
• Hawkes Processes on Graphons
• Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training