F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Abstract

Traditional centralized multi-agent reinforcement learning (MARL) algorithms are sometimes unpractical in complicated applications, due to non-interactivity between agents, curse of dimensionality and computation complexity. Hence, several decentralized MARL algorithms are motivated. However, existing decentralized methods only handle the fully cooperative setting where massive information needs to be transmitted in training. The block coordinate gradient descent scheme they used for successive independent actor and critic steps can simplify the calculation, but it causes serious bias. In this paper, we propose a flexible fully decentralized actor-critic MARL framework, which can combine most of actor-critic methods, and handle large-scale general cooperative multi-agent setting. A primal-dual hybrid gradient descent type algorithm framework is designed to learn individual agents separately for decentralization. From the perspective of each agent, policy improvement and value evaluation are jointly optimized, which can stabilize multi-agent policy learning. Furthermore, our framework can achieve scalability and stability for large-scale environment and reduce information transmission, by the parameter sharing mechanism and a novel modeling-other-agents methods based on theory-of-mind and online supervised learning. Sufficient experiments in cooperative Multi-agent Particle Environment and StarCraft II show that our decentralized MARL instantiation algorithms perform competitively against conventional centralized and decentralized methods.

arXiv comment75 pages, 10 figures, JMLR camera ready

References in corpus (24)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Reinforcement Learning with Deep Energy-Based Policies
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Mean Field Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Three Factors Influencing Minima in SGD
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Generalization and Regularization in DQN
• Machine Theory of Mind
• Multiagent Soft Q-Learning
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Learning to Coordinate Efficiently: A Model-based Approach
• Graph Convolutional Reinforcement Learning
• Regularization Matters in Policy Optimization
• Learning to Communicate Implicitly By Actions

Cited by in corpus (4)

• Social physics
• Multi-Agent Trust Region Policy Optimization
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning