Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

Abstract

We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information between their actions. Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols. The influence rewards for all agents can be computed in a decentralized way by enabling agents to learn a model of other agents using deep neural networks. In contrast, key previous works on emergent communication in the MARL setting were unable to learn diverse policies in a decentralized manner and had to resort to centralized training. Consequently, the influence reward opens up a window of new opportunities for research in this area.

Cited by in corpus (87)

• Emergent Tool Use From Multi-Agent Autocurricula
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• A survey on intrinsic motivation in reinforcement learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Socially-Compatible Behavior Design of Autonomous Vehicles with Verification on Real Human Data
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Towards Joint Learning of Optimal MAC Signaling and Wireless Channel Access
• Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control
• Emergent Multi-Agent Communication in the Deep Learning Era
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Graph Convolutional Reinforcement Learning
• On the interaction between supervision and self-play in emergent communication
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Auctions Between Regret-Minimizing Agents
• Open Problems in Cooperative AI
• Learning to Incentivize Other Learning Agents
• Ease-of-Teaching and Language Structure from Emergent Communication
• Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
• Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Learning Latent Representations to Influence Multi-Agent Interaction
• Learning Latent Representations to Co-Adapt to Humans
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
• Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• Emergent Language: A Survey and Taxonomy
• Provable Fictitious Play for General Mean-Field Games
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Learning to Communicate Implicitly By Actions
• Emergent Road Rules In Multi-Agent Driving Environments
• Cooperation and Reputation Dynamics with Reinforcement Learning
• Human Perception of Intrinsically Motivated Autonomy in Human-Robot Interaction
• Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
• Grounding Artificial Intelligence in the Origins of Human Behavior
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Learning to Draw: Emergent Communication through Sketching
• Learning to Communicate Using Counterfactual Reasoning
• MARL with General Utilities via Decentralized Shadow Reward Actor-Critic
• Steps Towards Satisficing Distributed Dynamic Team Trust
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Joint Attention for Multi-Agent Coordination and Social Learning
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Learning to Deceive in Multi-Agent Hidden Role Games
• Designed to Cooperate: A Kant-Inspired Ethic of Machine-to-Machine Cooperation
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• Disentangling causal effects for hierarchical reinforcement learning
• Correcting Experience Replay for Multi-Agent Communication
• Multi-Agent Reinforcement Learning and Human Social Factors in Climate Change Mitigation
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities
• Birds of a Feather Flock Together: A Close Look at Cooperation Emergence via Multi-Agent RL
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• REMAX: Relational Representation for Multi-Agent Exploration
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges
• Bounding the Inefficiency of Route Control in Intelligent Transport Systems
• "Conservatives Overfit, Liberals Underfit": The Social-Psychological Control of Affect and Uncertainty
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Multi-lingual agents through multi-headed neural networks
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Characterizing the Decidability of Finite State Automata Team Games with Communication
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Improving Social Welfare While Preserving Autonomy via a Pareto Mediator
• Reinforced Natural Language Interfaces via Entropy Decomposition
• Generation of Traffic Flows in Multi-Agent Traffic Simulation with Agent Behavior Model based on Deep Reinforcement Learning
• Improved cooperation by balancing exploration and exploitation in intertemporal social dilemma tasks
• Collaboration Promotes Group Resilience in Multi-Agent RL
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Spatial Influence-aware Reinforcement Learning for Intelligent Transportation System
• Explore and Control with Adversarial Surprise
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• Emergent Communication with World Models
• DSDF: An approach to handle stochastic agents in collaborative multi-agent reinforcement learning
• End-to-End Learning and Intervention in Games
• Improved Cooperation by Exploiting a Common Signal