Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations

Abstract

Multi-agent reinforcement learning systems aim to provide interacting agents with the ability to collaboratively learn and adapt to the behaviour of other agents. In many real-world applications, the agents can only acquire a partial view of the world. Here we consider a setting whereby most agents' observations are also extremely noisy, hence only weakly correlated to the true state of the environment. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we propose a multi-agent deep deterministic policy gradient algorithm enhanced by a communication medium (MADDPG-M), which implements a two-level, concurrent learning mechanism. An agent's policy depends on its own private observations as well as those explicitly shared by others through a communication medium. At any given point in time, an agent must decide whether its private observations are sufficiently informative to be shared with others. However, our environments provide no explicit feedback informing an agent whether a communication action is beneficial, rather the communication policies must also be learned through experience concurrently to the main policies. Our experimental results demonstrate that the algorithm performs well in six highly non-stationary environments of progressively higher complexity, and offers substantial performance gains compared to the baselines.

arXiv commentTo appear in Deep Reinforcement Learning Workshop, NIPS 2018

References in corpus (7)

• Categorical Reparameterization with Gumbel-Softmax
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Learning Multiagent Communication with Backpropagation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Unifying Count-Based Exploration and Intrinsic Motivation
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input

Cited by in corpus (8)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Learning Efficient Multi-agent Communication: An Information Bottleneck Approach
• Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning
• Attention-based Fault-tolerant Approach for Multi-agent Reinforcement Learning Systems