DORB: Dynamically Optimizing Multiple Rewards with Bandits

Abstract

Policy gradients-based reinforcement learning has proven to be a promising approach for directly optimizing non-differentiable evaluation metrics for language generation tasks. However, optimizing for a specific metric reward leads to improvements in mostly that metric only, suggesting that the model is gaming the formulation of that metric in a particular way without often achieving real qualitative improvements. Hence, it is more beneficial to make the model optimize multiple diverse metric rewards jointly. While appealing, this is challenging because one needs to manually decide the importance and scaling weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit approach (DORB), where at each round, the bandit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierarchical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our approaches via various automatic metrics and human evaluation on two important NLG tasks: question generation and data-to-text generation, including on an unseen-test transfer setup. Finally, we present interpretable analyses of the learned bandit curriculum over the optimized rewards.

arXiv commentEMNLP 2020 (15 pages)

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• A Neural Conversational Model
• A Deep Reinforced Model for Abstractive Summarization
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Deep Reinforcement Learning for Dialogue Generation
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Reinforcement Learning Neural Turing Machines - Revised
• Learning to Generate Questions by Learning What not to Generate
• A Survey of Online Experiment Design with the Stochastic Multi-Armed Bandit
• A Unified Query-based Generative Model for Question Generation and Question Answering
• Meta-Learning for Contextual Bandit Exploration
• A Bandit Framework for Optimal Selection of Reinforcement Learning Agents