Episodic Curiosity through Reachability

Abstract

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

arXiv commentAccepted to ICLR 2019. Code at https://github.com/google-research/episodic-curiosity/. Videos at https://sites.google.com/view/episodic-curiosity/

References in corpus (15)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• VIME: Variational Information Maximizing Exploration
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Exploration by Random Network Distillation
• Model-Free Episodic Control
• Automatic Goal Generation for Reinforcement Learning Agents
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Semi-parametric Topological Memory for Navigation
• Time-Contrastive Networks: Self-Supervised Learning from Video

Cited by in corpus (48)

• Exploration in Deep Reinforcement Learning: A Survey
• Agent57: Outperforming the Atari Human Benchmark
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• A survey on intrinsic motivation in reinforcement learning
• Never Give Up: Learning Directed Exploration Strategies
• Large-Scale Long-Tailed Recognition in an Open World
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• A Survey of Embodied AI: From Simulators to Research Tasks
• Action and Perception as Divergence Minimization
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• A Short Survey On Memory Based Reinforcement Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Learning Affordance Landscapes for Interaction Exploration in 3D Environments
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Learning to Visually Navigate in Photorealistic Environments Without any Supervision
• Geometric Entropic Exploration
• Bayesian Relational Memory for Semantic Visual Navigation
• Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Scaling Local Control to Large-Scale Topological Navigation
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• Parenting: Safe Reinforcement Learning from Human Input
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• Shaping embodied agent behavior with activity-context priors from egocentric video
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Touch-based Curiosity for Sparse-Reward Tasks
• Learning Efficient Multi-Agent Cooperative Visual Exploration
• Learning Diverse Policies with Soft Self-Generated Guidance
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Exploring Unknown States with Action Balance
• Learning the Arrow of Time
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Hard Attention Control By Mutual Information Maximization
• Building Intelligent Autonomous Navigation Agents
• Reconciling Spatial and Temporal Abstractions for Goal Representation
• ACDER: Augmented Curiosity-Driven Experience Replay