Inequity aversion improves cooperation in intertemporal social dilemmas

Abstract

Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.

arXiv comment15 pages, 8 figures

Cited by in corpus (19)

• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Intrinsic fluctuations of reinforcement learning promote cooperation
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Learning to Incentivize Other Learning Agents
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Generosity, selfishness and exploitation as optimal greedy strategies for resource sharing
• Grounding Artificial Intelligence in the Origins of Human Behavior
• Investigating the Impact of Direct Punishment on the Emergence of Cooperation in Multi-Agent Reinforcement Learning Systems
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• Influence-Based Multi-Agent Exploration
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Birds of a Feather Flock Together: A Close Look at Cooperation Emergence via Multi-Agent RL
• Improved cooperation by balancing exploration and exploitation in intertemporal social dilemma tasks
• Improved Cooperation by Exploiting a Common Signal
• Improving Social Welfare While Preserving Autonomy via a Pareto Mediator