Planning to Explore via Self-Supervised World Models

Abstract

Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at https://ramanans1.github.io/plan2explore/

Accepted at ICML 2020. Videos and code at https://ramanans1.github.io/plan2explore/

References in corpus (16)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• DeepMind Control Suite
• Learning Latent Dynamics for Planning from Pixels
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Exploration by Random Network Distillation
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Model-Based Active Exploration
• Dynamics-Aware Unsupervised Discovery of Skills
• Deep Dynamics Models for Learning Dexterous Manipulation
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Explicit Explore-Exploit Algorithms in Continuous State Spaces
• Ready Policy One: World Building Through Active Learning

Cited by in corpus (20)

• Self-Supervised Policy Adaptation during Deployment
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Reinforcement Learning with Prototypical Representations
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Action and Perception as Divergence Minimization
• On the model-based stochastic value gradient for continuous reinforcement learning
• Behavior From the Void: Unsupervised Active Pre-Training
• Evaluating Agents without Rewards
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Exploratory State Representation Learning
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Online Safety Assurance for Deep Reinforcement Learning
• Learning Compositional Neural Programs for Continuous Control
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Benchmarking the Spectrum of Agent Capabilities
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Low-Variance Policy Gradient Estimation with World Models