Ready Policy One: World Building Through Active Learning

Abstract

Model-Based Reinforcement Learning (MBRL) offers a promising direction for sample efficient learning, often achieving state of the art results for continuous control tasks. However, many existing MBRL methods rely on combining greedy policies with exploration heuristics, and even those which utilize principled exploration bonuses construct dual objectives in an ad hoc fashion. In this paper we introduce Ready Policy One (RP1), a framework that views MBRL as an active learning problem, where we aim to improve the world model in the fewest samples possible. RP1 achieves this by utilizing a hybrid objective function, which crucially adapts during optimization, allowing the algorithm to trade off reward v.s. exploration at different stages of learning. In addition, we introduce a principled mechanism to terminate sample collection once we have a rich enough trajectory batch to improve the model. We rigorously evaluate our method on a variety of continuous control tasks, and demonstrate statistically significant gains over existing approaches.

References in corpus (14)

• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Recurrent World Models Facilitate Policy Evolution
• Learning Latent Dynamics for Planning from Pixels
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Benchmarking Model-Based Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• When to Trust Your Model: Model-Based Policy Optimization
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Model-Based Active Exploration
• Deep Dynamics Models for Learning Dexterous Manipulation
• If MaxEnt RL is the Answer, What is the Question?
• Explicit Explore-Exploit Algorithms in Continuous State Spaces

Cited by in corpus (14)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Planning to Explore via Self-Supervised World Models
• Effective Diversity in Population Based Reinforcement Learning
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Active Exploration based on Information Gain by Particle Filter for Efficient Spatial Concept Formation
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Centralized Model and Exploration Policy for Multi-Agent RL
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Active Offline Policy Selection
• Customs Fraud Detection in the Presence of Concept Drift
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning