Multi-Agent Constrained Policy Optimisation

Abstract

Developing reinforcement learning algorithms that satisfy safety constraints is becoming increasingly important in real-world applications. In multi-agent reinforcement learning (MARL) settings, policy optimisation with safety awareness is particularly challenging because each individual agent has to not only meet its own safety constraints, but also consider those of others so that their joint behaviour can be guaranteed safe. Despite its importance, the problem of safe multi-agent learning has not been rigorously studied; very few solutions have been proposed, nor a sharable testing environment or benchmarks. To fill these gaps, in this work, we formulate the safe MARL problem as a constrained Markov game and solve it with policy optimisation methods. Our solutions -- Multi-Agent Constrained Policy Optimisation (MACPO) and MAPPO-Lagrangian -- leverage the theories from both constrained policy optimisation and multi-agent trust region learning. Crucially, our methods enjoy theoretical guarantees of both monotonic improvement in reward and satisfaction of safety constraints at every iteration. To examine the effectiveness of our methods, we develop the benchmark suite of Safe Multi-Agent MuJoCo that involves a variety of MARL baselines. Experimental results justify that MACPO/MAPPO-Lagrangian can consistently satisfy safety constraints, meanwhile achieving comparable performance to strong baselines.

References in corpus (13)

• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Game-Theoretic Multiagent Reinforcement Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• Safe Exploration in Markov Decision Processes
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
• MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shielding
• Settling the Variance of Multi-Agent Policy Gradients
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems

Cited by in corpus (2)

• Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand Management in Distribution Network
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges