Learning to Explore with Meta-Policy Gradient

Abstract

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore \emph{local} regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a \emph{global exploration} that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.

arXiv comment10 pages

References in corpus (12)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Learning to reinforcement learn
• Parameter Space Noise for Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Unifying Count-Based Exploration and Intrinsic Motivation

Cited by in corpus (11)

• Efficient Exploration via State Marginal Matching
• Single Episode Policy Transfer in Reinforcement Learning
• Experience Replay Optimization
• Meta-Learning for Contextual Bandit Exploration
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy