Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Abstract

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.

References in corpus (15)

• Counterfactual Multi-Agent Policy Gradients
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Guided Deep Reinforcement Learning for Swarm Systems
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning
• Emergent Social Learning via Multi-agent Reinforcement Learning
• Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning
• A Q-values Sharing Framework for Multiagent Reinforcement Learning under Budget Constraint
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search

Cited by in corpus (5)

• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• An Empirical Study on Google Research Football Multi-agent Scenarios
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Improving the Generalization of Unseen Crowd Behaviors for Reinforcement Learning based Local Motion Planners
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning