Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning

Abstract

Parameter sharing, where each agent independently learns a policy with fully shared parameters between all policies, is a popular baseline method for multi-agent deep reinforcement learning. Unfortunately, since all agents share the same policy network, they cannot learn different policies or tasks. This issue has been circumvented experimentally by adding an agent-specific indicator signal to observations, which we term "agent indication". Agent indication is limited, however, in that without modification it does not allow parameter sharing to be applied to environments where the action spaces and/or observation spaces are heterogeneous. This work formalizes the notion of agent indication and proves that it enables convergence to optimal policies for the first time. Next, we formally introduce methods to extend parameter sharing to learning in heterogeneous observation and action spaces, and prove that these methods allow for convergence to optimal policies. Finally, we experimentally confirm that the methods we introduce function empirically, and conduct a wide array of experiments studying the empirical efficacy of many different agent indication schemes for image based observation spaces.

References in corpus (6)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
• Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning
• SuperSuit: Simple Microwrappers for Reinforcement Learning Environments
• Multi-Agent Informational Learning Processes

Cited by in corpus (6)

• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning
• Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
• Multiplayer Support for the Arcade Learning Environment
• Agent Environment Cycle Games
• Scalable Multi-agent Reinforcement Learning Algorithm for Wireless Networks