Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net
arXiv:1909.08776
Abstract
In many real-world multi-robot tasks, high-quality solutions often require a team of robots to perform asynchronous actions under decentralized control. Decentralized multi-agent reinforcement learning methods have difficulty learning decentralized policies because of the environment appearing to be non-stationary due to other agents also learning at the same time. In this paper, we address this challenge by proposing a macro-action-based decentralized multi-agent double deep recurrent Q-net (MacDec-MADDRQN) which trains each decentralized Q-net using a centralized Q-net for action selection. A generalized version of MacDec-MADDRQN with two separate training environments, called Parallel-MacDec-MADDRQN, is also presented to leverage either centralized or decentralized exploration. The advantages and the practical nature of our methods are demonstrated by achieving near-centralized results in simulation and having real robots accomplish a warehouse tool delivery task in an efficient way.
References in corpus (6)
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Counterfactual Multi-Agent Policy Gradients
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability