Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices

Abstract

The goal of meta-reinforcement learning (meta-RL) is to build agents that can quickly learn new tasks by leveraging prior experience on related tasks. Learning a new task often requires both exploring to gather task-relevant information and exploiting this information to solve the task. In principle, optimal exploration and exploitation can be learned end-to-end by simply maximizing task performance. However, such meta-RL approaches struggle with local optima due to a chicken-and-egg problem: learning to explore requires good exploitation to gauge the exploration's utility, but learning to exploit requires information gathered via exploration. Optimizing separate objectives for exploration and exploitation can avoid this problem, but prior meta-RL exploration objectives yield suboptimal policies that gather information irrelevant to the task. We alleviate both concerns by constructing an exploitation objective that automatically identifies task-relevant information and an exploration objective to recover only this information. This avoids local optima in end-to-end training, without sacrificing optimal exploration. Empirically, DREAM substantially outperforms existing approaches on complex meta-RL problems, such as sparse-reward 3D visual navigation. Videos of DREAM: https://ezliu.github.io/dream/

arXiv commentInternational Conference on Machine Learning (ICML), 2021

References in corpus (30)

• Dueling Network Architectures for Deep Reinforcement Learning
• Theoretical Models of Learning to Learn
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Learning to reinforcement learn
• Deep Variational Information Bottleneck
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• One-shot Learning with Memory-Augmented Neural Networks
• Exploration by Random Network Distillation
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Variational Intrinsic Control
• Evolved Policy Gradients
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning to Generalize from Sparse and Underspecified Rewards
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta reinforcement learning as task inference
• Environment Probing Interaction Policies
• A Tutorial on Thompson Sampling
• Guided Meta-Policy Search
• Been There, Done That: Meta-Learning with Episodic Recall
• Meta-Q-Learning
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
• Offline Meta Learning of Exploration
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Meta-Model-Based Meta-Policy Optimization
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
• MAME : Model-Agnostic Meta-Exploration

Cited by in corpus (4)

• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem