Transfer Reward Learning for Policy Gradient-Based Text Generation

Abstract

Task-specific scores are often used to optimize for and evaluate the performance of conditional text generation systems. However, such scores are non-differentiable and cannot be used in the standard supervised learning paradigm. Hence, policy gradient methods are used since the gradient can be computed without requiring a differentiable objective. However, we argue that current n-gram overlap based measures that are used as rewards can be improved by using model-based rewards transferred from tasks that directly compare the similarity of sentence pairs. These reward models either output a score of sentence-level syntactic and semantic similarity between entire predicted and target sentences as the expected return, or for intermediate phrases as segmented accumulative rewards. We demonstrate that using a \textit{Transferable Reward Learner} leads to improved results on semantical evaluation measures in policy-gradient models for image captioning tasks. Our InferSent actor-critic model improves over a BLEU trained actor-critic model on MSCOCO when evaluated on a Word Mover's Distance similarity measure by 6.97 points, also improving on a Sliding Window Cosine Similarity measure by 10.48 points. Similar performance improvements are also obtained on the smaller Flickr-30k dataset, demonstrating the general applicability of the proposed transfer learning method.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Sequence Level Training with Recurrent Neural Networks
• Deep reinforcement learning from human preferences
• A large annotated corpus for learning natural language inference
• An Actor-Critic Algorithm for Sequence Prediction
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Actor-Critic Sequence Training for Image Captioning
• Attention Correctness in Neural Image Captioning
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML