UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
arXiv:2010.02974
Abstract
VDN and QMIX are two popular value-based algorithms for cooperative MARL that learn a centralized action value function as a monotonic mixing of per-agent utilities. While this enables easy decentralization of the learned policy, the restricted joint action value function can prevent them from solving tasks that require significant coordination between agents at a given timestep. We show that this problem can be overcome by improving the joint exploration of all agents during training. Specifically, we propose a novel MARL approach called Universal Value Exploration (UneVEn) that learns a set of related tasks simultaneously with a linear decomposition of universal successor features. With the policies of already solved related tasks, the joint exploration process of all agents can be improved to help them achieve better coordination. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
arXiv commentPublished at ICML 2021
References in corpus (22)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Dueling Network Architectures for Deep Reinforcement Learning
- Noisy Networks for Exploration
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Counterfactual Multi-Agent Policy Gradients
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- QPLEX: Duplex Dueling Multi-Agent Q-Learning
- Deep Successor Reinforcement Learning
- MAVEN: Multi-Agent Variational Exploration
- Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
- RODE: Learning Roles to Decompose Multi-Agent Tasks
- Multiagent Soft Q-Learning
- Deep Coordination Graphs
- Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning
- Visual Semantic Planning using Deep Successor Representations
- Universal Successor Features Approximators
- Fast Task Inference with Variational Intrinsic Successor Features
- Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
- Symmetry Learning for Function Approximation in Reinforcement Learning