Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents

Abstract

Discovering successful coordinated behaviors is a central challenge in Multi-Agent Reinforcement Learning (MARL) since it requires exploring a joint action space that grows exponentially with the number of agents. In this paper, we propose a mechanism for achieving sufficient exploration and coordination in a team of agents. Specifically, agents are rewarded for contributing to a more diversified team behavior by employing proper intrinsic motivation functions. To learn meaningful coordination protocols, we structure agents' interactions by introducing a novel framework, where at each timestep, an agent simulates counterfactual rollouts of its policy and, through a sequence of computations, assesses the gap between other agents' current behaviors and their targets. Actions that minimize the gap are considered highly influential and are rewarded. We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such as the StarCraft Multi-Agent Challenge. Our methods show significantly improved performances over different baselines across all tasks.

arXiv comment14 pages, 5 figures, 6 Tables

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Counterfactual Multi-Agent Policy Gradients
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• A survey on intrinsic motivation in reinforcement learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Learning to Incentivize Other Learning Agents
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Learning Novel Policies For Tasks
• Influence-Based Multi-Agent Exploration
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Coordination-driven learning in multi-agent problem spaces
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning