Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
arXiv:2005.05441
Abstract
Action and observation delays exist prevalently in the real-world cyber-physical systems which may pose challenges in reinforcement learning design. It is particularly an arduous task when handling multi-agent systems where the delay of one agent could spread to other agents. To resolve this problem, this paper proposes a novel framework to deal with delays as well as the non-stationary training issue of multi-agent tasks with model-free deep reinforcement learning. We formally define the Delay-Aware Markov Game that incorporates the delays of all agents in the environment. To solve Delay-Aware Markov Games, we apply centralized training and decentralized execution that allows agents to use extra information to ease the non-stationarity issue of the multi-agent systems during training, without the need of a centralized controller during execution. Experiments are conducted in multi-agent particle environments including cooperative communication, cooperative navigation, and competitive experiments. We also test the proposed algorithm in traffic scenarios that require coordination of all autonomous vehicles to show the practical value of delay-awareness. Results show that the proposed delay-aware multi-agent reinforcement learning algorithm greatly alleviates the performance degradation introduced by delay. Codes and demo videos are available at: https://github.com/baimingc/delay-aware-MARL.
References in corpus (15)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Categorical Reparameterization with Gumbel-Softmax
- Benchmarking Deep Reinforcement Learning for Continuous Control
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- CARLA: An Open Urban Driving Simulator
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
- Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
- At Human Speed: Deep Reinforcement Learning with Action Delay
- Reactive Reinforcement Learning in Asynchronous Environments
- Real-Time Reinforcement Learning
- Delay-Aware Model-Based Reinforcement Learning for Continuous Control