Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Abstract

We present a multi-agent actor-critic method that aims to implicitly address the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment among agents may not require an explicit formulation as long as (1) the policy gradients derived from a centralized critic carry sufficient information for the decentralized agents to maximize their joint action value through optimal cooperation and (2) a sustained level of exploration is enforced throughout training. Under the centralized training with decentralized execution (CTDE) paradigm, we achieve the former by formulating the centralized critic as a hypernetwork such that a latent state representation is integrated into the policy gradients through its multiplicative association with the stochastic policies; to achieve the latter, we derive a simple technique called adaptive entropy regularization where magnitudes of the entropy gradients are dynamically rescaled based on the current policy stochasticity to encourage consistent levels of exploration. Our algorithm, referred to as LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.

arXiv commentNeurIPS 2020 Camera Ready; first two authors contributed equally

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Counterfactual Multi-Agent Policy Gradients
• Reinforcement Learning with Deep Energy-Based Policies
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning Continuous Control Policies by Stochastic Value Gradients
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Memory-based control with recurrent neural networks
• Sample Efficient Actor-Critic with Experience Replay
• Equivalence Between Policy Gradients and Soft Q-Learning
• Guided Deep Reinforcement Learning for Swarm Systems
• MAVEN: Multi-Agent Variational Exploration
• If MaxEnt RL is the Answer, What is the Question?
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
• Reinforcement Learning by Value Gradients
• Credit Assignment Techniques in Stochastic Computation Graphs

Cited by in corpus (6)

• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents