Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
arXiv:2106.03400
Abstract
Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.
Accepted by NeurIPS2021. The first two authors contributed equally to the work
References in corpus (9)
- Conservative Q-Learning for Offline Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
- Off-Policy Multi-Agent Decomposed Policy Gradients
- Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning