Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
arXiv:2006.07781
Abstract
Conventional Reinforcement Learning (RL) algorithms usually have one single agent learning to solve the task independently. As a result, the agent can only explore a limited part of the state-action space while the learned behavior is highly correlated to the agent's previous experience, making the training prone to a local minimum. In this work, we empower RL with the capability of teamwork and propose a novel non-local policy optimization framework called Diversity-regularized Collaborative Exploration (DiCE). DiCE utilizes a group of heterogeneous agents to explore the environment simultaneously and share the collected experiences. A regularization mechanism is further designed to maintain the diversity of the team and modulate the exploration. We implement the framework in both on-policy and off-policy settings and the experimental results show that DiCE can achieve substantial improvement over the baselines in the MuJoCo locomotion tasks.
https://decisionforce.github.io/DiCE/
References in corpus (17)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Noisy Networks for Exploration
- VIME: Variational Information Maximizing Exploration
- Parameter Space Noise for Exploration
- Large-Scale Study of Curiosity-Driven Learning
- Count-Based Exploration with Neural Density Models
- Distributed Distributional Deterministic Policy Gradients
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Exploration by Random Network Distillation
- Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
- Better Exploration with Optimistic Actor-Critic
- Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
- Learning Novel Policies For Tasks