CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

Abstract

In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.

Accepted at ICML 2019 https://github.com/flowersteam/curious

References in corpus (22)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Hindsight Experience Replay
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Data-Efficient Hierarchical Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Distral: Robust Multitask Reinforcement Learning
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner
• Hierarchical Reinforcement Learning with Hindsight
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Many-Goals Reinforcement Learning
• Energy-Based Hindsight Experience Prioritization
• Multi-task Deep Reinforcement Learning with PopArt

Cited by in corpus (12)

• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Efficient Exploration via State Marginal Matching
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces
• Hierarchical Affordance Discovery using Intrinsic Motivation
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Intrinsically Motivated Open-Ended Multi-Task Learning Using Transfer Learning to Discover Task Hierarchy
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• Interleaved Multitask Learning with Energy Modulated Learning Progress
• Autonomous Goal Exploration using Learned Goal Spaces for Visuomotor Skill Acquisition in Robots