On Memory Mechanism in Multi-Agent Reinforcement Learning

Abstract

Multi-agent reinforcement learning (MARL) extends (single-agent) reinforcement learning (RL) by introducing additional agents and (potentially) partial observability of the environment. Consequently, algorithms for solving MARL problems incorporate various extensions beyond traditional RL methods, such as a learned communication protocol between cooperative agents that enables exchange of private information or adaptive modeling of opponents in competitive settings. One popular algorithmic construct is a memory mechanism such that an agent's decisions can depend not only upon the current state but also upon the history of observed states and actions. In this paper, we study how a memory mechanism can be useful in environments with different properties, such as observability, internality and presence of a communication channel. Using both prior work and new experiments, we show that a memory mechanism is helpful when learning agents need to model other agents and/or when communication is constrained in some way; however we must to be cautious of agents achieving effective memoryfulness through other means.

References in corpus (16)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Measuring Catastrophic Forgetting in Neural Networks
• CycleGAN, a Master of Steganography
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Machine Theory of Mind
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• R-MADDPG for Partially Observable Environments and Limited Communication
• Compositional Obverter Communication Learning From Raw Visual Input
• Emergence of Communication in an Interactive World with Consistent Speakers
• Emergent Communication in a Multi-Modal, Multi-Step Referential Game
• A Paradigm for Situated and Goal-Driven Language Learning