Counterfactual Multi-Agent Policy Gradients

Abstract

Cooperative multi-agent systems can be naturally used to model many real world problems, such as network packet routing and the coordination of autonomous vehicles. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Cited by in corpus (218)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Mean Field Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Game-Theoretic Multiagent Reinforcement Learning
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics
• R-MADDPG for Partially Observable Environments and Limited Communication
• Multi-agent Reinforcement Learning for Networked System Control
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Multi-Agent Common Knowledge Reinforcement Learning
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Deep Coordination Graphs
• Off-Policy Multi-Agent Decomposed Policy Gradients
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Optimizing Collision Avoidance in Dense Airspace using Deep Reinforcement Learning
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Graph Convolutional Reinforcement Learning
• "Other-Play" for Zero-Shot Coordination
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance
• TarMAC: Targeted Multi-Agent Communication
• Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization
• A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
• Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks
• Settling the Variance of Multi-Agent Policy Gradients
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
• A Generalized Training Approach for Multiagent Learning
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Graph Policy Gradients for Large Scale Robot Control
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Collaborative Visual Navigation
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Learning to Incentivize Other Learning Agents
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• Attend2Pack: Bin Packing through Deep Reinforcement Learning with Attention
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Interaction-aware Decision Making with Adaptive Strategies under Merging Scenarios
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Learning Latent Representations to Influence Multi-Agent Interaction
• Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
• Difference Rewards Policy Gradients
• On Multi-Agent Learning in Team Sports Games
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
• Neural Replicator Dynamics
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
• Learning Strategies in Decentralized Matching Markets under Uncertain Preferences
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
• Cooperative Multi-Agent Reinforcement Learning with Partial Observations
• Efficient Collaborative Multi-Agent Deep Reinforcement Learning for Large-Scale Fleet Management
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• A Deep Reinforcement Learning Approach for Traffic Signal Control Optimization
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic
• Privileged Information Dropout in Reinforcement Learning
• Fast Sequence Generation with Multi-Agent Reinforcement Learning
• Dynamic Queue-Jump Lane for Emergency Vehicles under Partially Connected Settings: A Multi-Agent Deep Reinforcement Learning Approach
• Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• Multi-Agent Determinantal Q-Learning
• Context-Aware Sparse Deep Coordination Graphs
• Incorporating Pragmatic Reasoning Communication into Emergent Language
• Dynamic Causal Bayesian Optimization
• Distributed Reinforcement Learning for Cooperative Multi-Robot Object Manipulation
• Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
• Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios
• Distributed Heuristic Multi-Agent Path Finding with Communication
• Efficient Ridesharing Dispatch Using Multi-Agent Reinforcement Learning
• Value-Decomposition Multi-Agent Actor-Critics
• Learning to Communicate Using Counterfactual Reasoning
• XDO: A Double Oracle Algorithm for Extensive-Form Games
• Influence-Based Multi-Agent Exploration
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Altruistic Maneuver Planning for Cooperative Autonomous Vehicles Using Multi-agent Advantage Actor-Critic
• Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Coordination in Adversarial Sequential Team Games via Multi-Agent Deep Reinforcement Learning
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning
• Independent Natural Policy Gradient Always Converges in Markov Potential Games
• Heterogeneous Graph Attention Networks for Learning Diverse Communication
• Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search
• Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• Centralized Model and Exploration Policy for Multi-Agent RL
• Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach
• Offline Decentralized Multi-Agent Reinforcement Learning
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• Optimizing Large-Scale Fleet Management on a Road Network using Multi-Agent Deep Reinforcement Learning with Graph Neural Network
• Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks
• Arena: a toolkit for Multi-Agent Reinforcement Learning
• Correcting Experience Replay for Multi-Agent Communication
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Multi-Agent Interactions Modeling with Correlated Policies
• Diverse Behavior Is What Game AI Needs: Generating Varied Human-Like Playing Styles Using Evolutionary Multi-Objective Deep Reinforcement Learning
• Skill Discovery of Coordination in Multi-agent Reinforcement Learning
• Competitive Multi-Agent Deep Reinforcement Learning with Counterfactual Thinking
• Defensive Escort Teams via Multi-Agent Deep Reinforcement Learning
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweeping
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Human and Multi-Agent collaboration in a human-MARL teaming framework
• Competing AI: How does competition feedback affect machine learning?
• Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation
• Time-series Imputation of Temporally-occluded Multiagent Trajectories
• Who2com: Collaborative Perception via Learnable Handshake Communication
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Decentralized Reinforcement Learning for Multi-Target Search and Detection by a Team of Drones
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Robust Temporal Difference Learning for Critical Domains
• Mimicking Evolution with Reinforcement Learning
• Distributed Policy Iteration for Scalable Approximation of Cooperative Multi-Agent Policies
• Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net
• Competitive Policy Optimization
• On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Decentralized Multi-Agent Actor-Critic with Generative Inference
• Inferring Personalized Bayesian Embeddings for Learning from Heterogeneous Demonstration
• Divergence-Regularized Multi-Agent Actor-Critic
• Efficient Multi-robot Exploration via Multi-head Attention-based Cooperation Strategy
• Multi-Robot Formation Control Using Reinforcement Learning
• Represented Value Function Approach for Large Scale Multi Agent Reinforcement Learning
• Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks
• Learning Complex Multi-Agent Policies in Presence of an Adversary
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph
• Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving
• Intelligent Link Adaptation for Grant-Free Access Cellular Networks: A Distributed Deep Reinforcement Learning Approach
• Local Patch AutoAugment with Multi-Agent Collaboration
• Many Agent Reinforcement Learning Under Partial Observability
• Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
• Coordinated Proximal Policy Optimization
• Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus
• KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• MAPEL: Multi-Agent Pursuer-Evader Learning using Situation Report
• Distributed interference cancellation in multi-agent scenarios
• Meta Reinforcement Learning Based Sensor Scanning in 3D Uncertain Environments for Heterogeneous Multi-Robot Systems
• Calculus of Consent via MARL: Legitimating the Collaborative Governance Supplying Public Goods
• Resilient Consensus-based Multi-agent Reinforcement Learning with Function Approximation
• Two-stage training algorithm for AI robot soccer
• Combining Propositional Logic Based Decision Diagrams with Decision Making in Urban Systems
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Token Manipulation Generative Adversarial Network for Text Generation
• Learning to Coordinate via Multiple Graph Neural Networks
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• Batch-Augmented Multi-Agent Reinforcement Learning for Efficient Traffic Signal Optimization
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• DSDF: An approach to handle stochastic agents in collaborative multi-agent reinforcement learning
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• Improved cooperation by balancing exploration and exploitation in intertemporal social dilemma tasks
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• On Solving Cooperative MARL Problems with a Few Good Experiences
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• Behaviour-conditioned policies for cooperative reinforcement learning tasks
• Attention-based Fault-tolerant Approach for Multi-agent Reinforcement Learning Systems