The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games

Abstract

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at \url{https://github.com/marlbenchmark/on-policy}.

arXiv commentThis paper has been accepted by NeurIPS 2022 Datasets and Benchmarks

Cited by in corpus (27)

• Energy Efficient Computation Offloading in Aerial Edge Networks With Multi-Agent Cooperation
• Adaptive Digital Twin for UAV-Assisted Integrated Sensing, Communication, and Computation Networks
• Theory of Mind for Multi-Agent Collaboration via Large Language Models
• Robust Multi-agent Communication via Multi-view Message Certification
• Human-Centric Resource Allocation for the Metaverse With Multiaccess Edge Computing
• Composite Motion Learning with Task Control
• AI-Driven Day-to-Day Route Choice
• Online Planning for Multi-UAV Pursuit-Evasion in Unknown Environments Using Deep Reinforcement Learning
• Enhancing the Robustness of QMIX against State-adversarial Attacks
• Toward multi-target self-organizing pursuit in a partially observable Markov game
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• Adaptive trajectory-constrained exploration strategy for deep reinforcement learning
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking
• Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning
• Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation
• Novel Multi-Agent Action Masked Deep Reinforcement Learning for General Industrial Assembly Lines Balancing Problems
• Graph Attention Multi-Agent Fleet Autonomy for Advanced Air Mobility
• Dynamic Co-Optimization Compiler: Leveraging Multi-Agent Reinforcement Learning for Enhanced DNN Accelerator Performance
• Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space
• Strategy Game-Playing with Size-Constrained State Abstraction
• Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
• IRL for Restless Multi-Armed Bandits with Applications in Maternal and Child Health
• An End-to-End Collaborative Learning Approach for Connected Autonomous Vehicles in Occluded Scenarios
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
• Cooperative and Asynchronous Transformer-based Mission Planning for Heterogeneous Teams of Mobile Robots
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi