Learning to Generalize from Sparse and Underspecified Rewards

Abstract

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often underspecified: they do not distinguish between purposeful and accidental success. Generalization from underspecified rewards hinges on discounting spurious trajectories that attain accidental success, while learning from sparse feedback requires effective exploration. We address exploration by using a mode covering direction of KL divergence to collect a diverse set of successful trajectories, followed by a mode seeking KL divergence to train a robust policy. We propose Meta Reward Learning (MeRL) to construct an auxiliary reward function that provides more refined feedback for learning. The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy. The MeRL approach outperforms our alternative reward learning technique based on Bayesian Optimization, and achieves the state-of-the-art on weakly-supervised semantic parsing. It improves previous work by 1.2% and 2.4% on WikiTableQuestions and WikiSQL datasets respectively.

ICML 2019

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• VQA: Visual Question Answering
• Learning to Reweight Examples for Robust Deep Learning
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• A Study on Overfitting in Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Meta-Gradient Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Grounded Language Learning in a Simulated 3D World
• Scalable agent alignment via reward modeling: a research direction
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Learning to Teach with Dynamic Loss Functions
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Learning to Understand Goal Specifications by Modelling Reward
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Reward Shaping via Meta-Learning
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Neural Program Synthesis with Priority Queue Training
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Natural Language to Structured Query Generation via Meta-Learning
• Adversarial TableQA: Attention Supervision for Question Answering on Tables
• Multi-task Maximum Entropy Inverse Reinforcement Learning
• Inferring Logical Forms From Denotations
• Macro Grammars and Holistic Triggering for Efficient Semantic Parsing
• It was the training data pruning too!

Cited by in corpus (21)

• TAPAS: Weakly Supervised Table Parsing via Pre-training
• Self-Supervised Generalisation with Meta Auxiliary Learning
• TAPEX: Table Pre-training via Learning a Neural SQL Executor
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• Guiding Deep Molecular Optimization with Genetic Exploration
• A Discrete Hard EM Approach for Weakly Supervised Question Answering
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Knowledge Injection into Dialogue Generation via Language Models
• Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
• Learning a natural-language to LTL executable semantic parser for grounded robotics
• SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
• Learning by Fixing: Solving Math Word Problems with Weak Supervision
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Self-supervised Text-to-SQL Learning with Header Alignment Training
• Answering Conversational Questions on Structured Data without Logical Forms
• Sentiment Analysis for Reinforcement Learning
• SPARQLing Database Queries from Intermediate Question Decompositions