Learning with AMIGo: Adversarially Motivated Intrinsic Goals

Abstract

A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating -- as form of meta-learning -- a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns to propose increasingly challenging -- yet achievable -- goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.

arXiv comment18 pages, 6 figures, published at The Ninth International Conference on Learning Representations (2021)

References in corpus (22)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Count-Based Exploration with Neural Density Models
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Episodic Curiosity through Reachability
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Learning to Understand Goal Specifications by Modelling Reward
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• The NetHack Learning Environment
• Automatic Curriculum Learning through Value Disagreement
• Automated curricula through setter-solver interactions
• TorchBeast: A PyTorch Platform for Distributed RL
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Self-Paced Contextual Reinforcement Learning
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents
• Self-Paced Deep Reinforcement Learning

Cited by in corpus (11)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• BeBold: Exploration Beyond the Boundary of Explored Regions
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• ELLA: Exploration through Learned Language Abstraction
• Learning Continuous Environment Fields via Implicit Functions
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Adversarial Environment Generation for Learning to Navigate the Web
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Selection for short-term empowerment accelerates the evolution of homeostatic neural cellular automata
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals