Correcting Experience Replay for Multi-Agent Communication

Abstract

We consider the problem of learning to communicate using multi-agent reinforcement learning (MARL). A common approach is to learn off-policy, using data sampled from a replay buffer. However, messages received in the past may not accurately reflect the current communication policy of each agent, and this complicates learning. We therefore introduce a 'communication correction' which accounts for the non-stationarity of observed communication induced by multi-agent learning. It works by relabelling the received message to make it likely under the communicator's current policy, and thus be a better reflection of the receiver's current environment. To account for cases in which agents are both senders and receivers, we introduce an ordered relabelling scheme. Our correction is computationally efficient and can be integrated with a range of off-policy algorithms. We find in our experiments that it substantially improves the ability of communicating MARL systems to learn across a variety of cooperative and competitive tasks.

References in corpus (21)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Prioritized Experience Replay
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Deep Recurrent Q-Learning for Partially Observable MDPs
• StarCraft II: A New Challenge for Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Multiagent Soft Q-Learning
• Hierarchical Reinforcement Learning with Hindsight
• Biases for Emergent Communication in Multi-agent Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL

Cited by in corpus (1)

• MARL with General Utilities via Decentralized Shadow Reward Actor-Critic