Actor-Attention-Critic for Multi-Agent Reinforcement Learning

Abstract

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

arXiv commentICML 2019 camera ready version

References in corpus (13)

• Continuous control with deep reinforcement learning
• Categorical Reparameterization with Gumbel-Softmax
• Trust Region Policy Optimization
• A Structured Self-attentive Sentence Embedding
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Recurrent Models of Visual Attention
• Emergence of Locomotion Behaviours in Rich Environments
• Counterfactual Multi-Agent Policy Gradients
• Learning Attentional Communication for Multi-Agent Cooperation
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Control of Memory, Active Perception, and Action in Minecraft
• Multiagent Soft Q-Learning
• Multi-focus Attention Network for Efficient Deep Reinforcement Learning

Cited by in corpus (23)

• A Review of Deep Reinforcement Learning for Smart Building Energy Management
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communication
• SACHA: Soft Actor-Critic with Heuristic-Based Attention for Partially Observable Multi-Agent Path Finding
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Joint Sensing, Communication, and Computation Resource Allocation for Cooperative Perception in Fog-Based Vehicular Networks
• KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Learning Multi-Agent Coordination through Connectivity-driven Communication
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization
• Set-to-Sequence Methods in Machine Learning: a Review
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
• Learning to Deceive in Multi-Agent Hidden Role Games
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Causal Mean Field Multi-Agent Reinforcement Learning