Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
arXiv:2006.07169
Abstract
Exploration in multi-agent reinforcement learning is a challenging problem, especially in environments with sparse rewards. We propose a general method for efficient exploration by sharing experience amongst agents. Our proposed algorithm, called Shared Experience Actor-Critic (SEAC), applies experience sharing in an actor-critic framework. We evaluate SEAC in a collection of sparse-reward multi-agent environments and find that it consistently outperforms two baselines and two state-of-the-art algorithms by learning in fewer steps and converging to higher returns. In some harder environments, experience sharing makes the difference between learning to solve the task and not learning at all.
arXiv commentPublished in 34th Conference on Neural Information Processing Systems (NeurIPS), see https://proceedings.neurips.cc/paper/2020/hash/7967cc8e3ab559e68cc944c44b1cf3e8-Abstract.html - This updated version of the paper is identical to the original paper published at NeurIPS 2020 but includes minor clarifications following recommendations in http://agents.inf.ed.ac.uk/blog/multiagent-rl-inaccuracies/
References in corpus (9)
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Counterfactual Multi-Agent Policy Gradients
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
- Deep Reinforcement Learning and the Deadly Triad
- Reinforcement Learning from Imperfect Demonstrations
- MAVEN: Multi-Agent Variational Exploration
- Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning