ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning

Abstract

Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failed experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, and even small amount of advice is sufficient for the agent to achieve good performance.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• A large annotated corpus for learning natural language inference
• Hindsight Experience Replay
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Learning to Understand Goal Specifications by Modelling Reward
• Beating Atari with Natural Language Guided Reinforcement Learning
• Teaching Machines to Describe Images via Natural Language Feedback

Cited by in corpus (7)

• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Interactive Learning from Activity Description
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy