Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Abstract

Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning. In the single-agent setting, this challenge is addressed by introducing intrinsic rewards that motivate agents to explore unseen regions of their state spaces; however, applying these techniques naively to the multi-agent setting results in agents exploring independently, without any coordination among themselves. Exploration in cooperative multi-agent settings can be accelerated and improved if agents coordinate their exploration. In this paper we introduce a framework for designing intrinsic rewards which consider what other agents have explored such that the agents can coordinate. Then, we develop an approach for learning how to dynamically select between several exploration modalities to maximize extrinsic rewards. Concretely, we formulate the approach as a hierarchical policy where a high-level controller selects among sets of policies trained on diverse intrinsic rewards and the low-level controllers learn the action policies of all agents under these specific rewards. We demonstrate the effectiveness of the proposed approach in cooperative domains with sparse rewards where state-of-the-art methods fail and challenging multi-stage tasks that necessitate changing modes of coordination.

References in corpus (15)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• VIME: Variational Information Maximizing Exploration
• Counterfactual Multi-Agent Policy Gradients
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Learning Multiagent Communication with Backpropagation
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Go-Explore: a New Approach for Hard-Exploration Problems
• MAVEN: Multi-Agent Variational Exploration

Cited by in corpus (9)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• A survey on intrinsic motivation in reinforcement learning
• Influencing Towards Stable Multi-Agent Interactions
• Influence-Based Multi-Agent Exploration
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Cooperative and Competitive Biases for Multi-Agent Reinforcement Learning
• REMAX: Relational Representation for Multi-Agent Exploration
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• A Bayesian Approach to Identifying Representational Errors