ELLA: Exploration through Learned Language Abstraction
arXiv:2103.05825
Abstract
Building agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these agents via reinforcement learning in environments with synthetic language; however, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex BabyAI environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods.
arXiv comment19 pages, 9 figures. Published in Conference on Neural Information Processing Systems (NeurIPS) 2021. Appendix includes supplementary experiments
References in corpus (13)
- Large-Scale Study of Curiosity-Driven Learning
- Hindsight Experience Replay
- Modular Multitask Reinforcement Learning with Policy Sketches
- Grounded Language Learning in a Simulated 3D World
- Language as an Abstraction for Hierarchical Deep Reinforcement Learning
- Explaining Classifiers with Causal Concept Effect (CaCE)
- From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- Learning to Understand Goal Specifications by Modelling Reward
- Learning with AMIGo: Adversarially Motivated Intrinsic Goals
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
- A Narration-based Reward Shaping Approach using Grounded Natural Language Commands