Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration

Abstract

Many reality tasks such as robot coordination can be naturally modelled as multi-agent cooperative system where the rewards are sparse. This paper focuses on learning decentralized policies for such tasks using sub-optimal demonstration. To learn the multi-agent cooperation effectively and tackle the sub-optimality of demonstration, a self-improving learning method is proposed: On the one hand, the centralized state-action values are initialized by the demonstration and updated by the learned decentralized policy to improve the sub-optimality. On the other hand, the Nash Equilibrium are found by the current state-action value and are used as a guide to learn the policy. The proposed method is evaluated on the combat RTS games which requires a high level of multi-agent cooperation. Extensive experimental results on various combat scenarios demonstrate that the proposed method can learn multi-agent cooperation effectively. It significantly outperforms many state-of-the-art demonstration based approaches.

References in corpus (12)

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Generative Adversarial Imitation Learning
• Mean Field Multi-Agent Reinforcement Learning
• Learning Multiagent Communication with Backpropagation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Multi-agent Inverse Reinforcement Learning for Two-person Zero-sum Games
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations