CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

Abstract

A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.

Published at International Conference on Learning Representations 2020

References in corpus (15)

• Continuous control with deep reinforcement learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• The Communicative Multiagent Team Decision Problem: Analyzing Teamwork Theories and Models
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• FeUdal Networks for Hierarchical Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Emergent Coordination Through Competition
• Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
• Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization
• A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning
• M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning

Cited by in corpus (9)

• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Safe Multi-Agent Reinforcement Learning via Shielding
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Interaction-aware Decision Making with Adaptive Strategies under Merging Scenarios
• SLAS: Speed and Lane Advisory System for Highway Navigation
• On Multi-Agent Learning in Team Sports Games
• Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios
• Learning Efficient Multi-Agent Cooperative Visual Exploration
• Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships