Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Abstract

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

References in corpus (3)

• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Learning to Protect Communications with Adversarial Neural Cryptography
• Learning Conventions in Multiagent Stochastic Domains using Likelihood Estimates

Cited by in corpus (228)

• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Mean Field Multi-Agent Reinforcement Learning
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• A Survey of Deep Reinforcement Learning in Video Games
• Emergent Complexity via Multi-Agent Competition
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• Energy Management Based on Multi-Agent Deep Reinforcement Learning for A Multi-Energy Industrial Park
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Spatial Action Maps for Mobile Manipulation
• Reward Design for Driver Repositioning Using Multi-Agent Reinforcement Learning
• Shapley Q-value: A Local Reward Approach to Solve Global Reward Games
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• R-MADDPG for Partially Observable Environments and Limited Communication
• Learning to Schedule Communication in Multi-agent Reinforcement Learning
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Multiagent Soft Q-Learning
• Multi-Agent Common Knowledge Reinforcement Learning
• Emergent Translation in Multi-Agent Communication
• Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communication
• Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• A Deep Policy Inference Q-Network for Multi-Agent Systems
• Deep Coordination Graphs
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems
• Robust Multi-agent Communication via Multi-view Message Certification
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• Factorized Q-Learning for Large-Scale Multi-Agent Systems
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Deep Multi-Agent Reinforcement Learning with Relevance Graphs
• Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Emergent Multi-Agent Communication in the Deep Learning Era
• Graph Convolutional Reinforcement Learning
• "Other-Play" for Zero-Shot Coordination
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Safe Multi-Agent Reinforcement Learning via Shielding
• Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys
• MADRaS : Multi Agent Driving Simulator
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Q-value Path Decomposition for Deep Multiagent Reinforcement Learning
• Sustainability of Data Center Digital Twins with Reinforcement Learning
• Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks
• A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Graph Policy Gradients for Large Scale Robot Control
• Consequentialist conditional cooperation in social dilemmas with imperfect information
• Accelerating Quadratic Optimization with Reinforcement Learning
• Multi-Agent Game Abstraction via Graph Attention Neural Network
• Distributed multi-agent target search and tracking with Gaussian process and reinforcement learning
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
• Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Towards Cooperation in Sequential Prisoner's Dilemmas: a Deep Multiagent Reinforcement Learning Approach
• KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning
• Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Learning Multi-Agent Coordination through Connectivity-driven Communication
• Interaction-aware Decision Making with Adaptive Strategies under Merging Scenarios
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Heterogeneous Multi-Agent Reinforcement Learning for Unknown Environment Mapping
• Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Learning Latent Representations to Influence Multi-Agent Interaction
• Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functions
• On Multi-Agent Learning in Team Sports Games
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Learning Latent Representations to Co-Adapt to Humans
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• Microscopic Traffic Simulation by Cooperative Multi-agent Deep Reinforcement Learning
• Neural Replicator Dynamics
• V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
• STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Networked Multi-Agent Reinforcement Learning with Emergent Communication
• Compositional Reinforcement Learning from Logical Specifications
• Emergence of Theory of Mind Collaboration in Multiagent Systems
• Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces
• Multi-Agent Reinforcement Learning with Multi-Step Generative Models
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• Macro-Action-Based Deep Multi-Agent Reinforcement Learning
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Cooperative Multi-Agent Reinforcement Learning with Partial Observations
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
• Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Games
• A Multi-Agent Reinforcement Learning Method for Impression Allocation in Online Display Advertising
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Reinforcement Learning Your Way: Agent Characterization through Policy Regularization
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Trajectory Design for UAV-Based Internet-of-Things Data Collection: A Deep Reinforcement Learning Approach
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Learning from My Partner's Actions: Roles in Decentralized Robot Teams
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• Learning Deep Mean Field Games for Modeling Large Population Behavior
• Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning
• A Framework for Studying Reinforcement Learning and Sim-to-Real in Robot Soccer
• Learning to Communicate Implicitly By Actions
• Actor Critic with Differentially Private Critic
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• A Regularized Opponent Model with Maximum Entropy Objective
• A Multi-Agent Deep Reinforcement Learning based Spectrum Allocation Framework for D2D Communications
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Application of Self-Play Reinforcement Learning to a Four-Player Game of Imperfect Information
• Reinforcement Communication Learning in Different Social Network Structures
• Learning to Communicate Using Counterfactual Reasoning
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Multiplayer Support for the Arcade Learning Environment
• Value-Decomposition Multi-Agent Actor-Critics
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• Visual Adversarial Imitation Learning using Variational Models
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
• XDO: A Double Oracle Algorithm for Extensive-Form Games
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• A New One-Point Residual-Feedback Oracle For Black-Box Learning and Control
• IntelligentCrowd: Mobile Crowdsensing via Multi-Agent Reinforcement Learning
• Joint Attention for Multi-Agent Coordination and Social Learning
• Altruistic Maneuver Planning for Cooperative Autonomous Vehicles Using Multi-agent Advantage Actor-Critic
• Non-cooperative Multi-agent Systems with Exploring Agents
• Multi-Agent Deep Reinforcement Learning based Spectrum Allocation for D2D Underlay Communications
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Learning and Management for Internet-of-Things: Accounting for Adaptivity and Scalability
• Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problems
• Evolving Self-supervised Neural Networks: Autonomous Intelligence from Evolved Self-teaching
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Robust Multi-agent Counterfactual Prediction
• Adaptive Synthetic Characters for Military Training
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach
• Game Theory and Machine Learning in UAVs-Assisted Wireless Communication Networks: A Survey
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• A Conceptual Bio-Inspired Framework for the Evolution of Artificial General Intelligence
• Decentralized Multi-Agent Reinforcement Learning for Task Offloading Under Uncertainty
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Fully Decentralized Reinforcement Learning-based Control of Photovoltaics in Distribution Grids for Joint Provision of Real and Reactive Power
• Centralized Model and Exploration Policy for Multi-Agent RL
• Shaping Advice in Deep Multi-Agent Reinforcement Learning
• Optimizing Large-Scale Fleet Management on a Road Network using Multi-Agent Deep Reinforcement Learning with Graph Neural Network
• Decentralized Reinforcement Learning for Multi-Target Search and Detection by a Team of Drones
• Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Graph Attention Multi-Agent Fleet Autonomy for Advanced Air Mobility
• Who2com: Collaborative Perception via Learnable Handshake Communication
• Multi-agent Cooperative Games Using Belief Map Assisted Training
• Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net
• Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks
• A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning
• The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning
• Efficient Multi-robot Exploration via Multi-head Attention-based Cooperation Strategy
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Learning through Probing: a decentralized reinforcement learning architecture for social dilemmas
• Multi-Robot Formation Control Using Reinforcement Learning
• Represented Value Function Approach for Large Scale Multi Agent Reinforcement Learning
• Multi-vehicle Flocking Control with Deep Deterministic Policy Gradient Method
• A coevolutionary approach to deep multi-agent reinforcement learning
• Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games
• Tactical Reward Shaping: Bypassing Reinforcement Learning with Strategy-Based Goals
• KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning
• Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning
• The AI Arena: A Framework for Distributed Multi-Agent Reinforcement Learning
• End-to-end Decentralized Multi-robot Navigation in Unknown Complex Environments via Deep Reinforcement Learning
• Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control
• Proficiency Constrained Multi-Agent Reinforcement Learning for Environment-Adaptive Multi UAV-UGV Teaming
• PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems
• Battlesnake Challenge: A Multi-agent Reinforcement Learning Playground with Human-in-the-loop
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Independent Reinforcement Learning for Weakly Cooperative Multiagent Traffic Control Problem
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• AGENT: A Benchmark for Core Psychological Reasoning
• Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving
• Prioritized Guidance for Efficient Multi-Agent Reinforcement Learning Exploration
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Many Agent Reinforcement Learning Under Partial Observability
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
• Agent Probing Interaction Policies
• General Automatic Solution Generation of Social Problems
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control
• Searching with Opponent-Awareness
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning
• RLCache: Automated Cache Management Using Reinforcement Learning
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Efficient Use of heuristics for accelerating XCS-based Policy Learning in Markov Games
• Reinforcement Learning for Volt-Var Control: A Novel Two-stage Progressive Training Strategy
• Generative Exploration and Exploitation
• MAPEL: Multi-Agent Pursuer-Evader Learning using Situation Report
• A Hybrid Algorithm for Metaheuristic Optimization
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
• Reinforcement Learning-driven Information Seeking: A Quantum Probabilistic Approach
• Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search
• Attention-based Fault-tolerant Approach for Multi-agent Reinforcement Learning Systems
• Meta Reinforcement Learning Based Sensor Scanning in 3D Uncertain Environments for Heterogeneous Multi-Robot Systems
• Improving On-policy Learning with Statistical Reward Accumulation