Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Abstract

Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation.

10 pages, 7 figures

References in corpus (2)

• Value Function Approximation in Zero-Sum Markov Games
• Empirically Evaluating Multiagent Learning Algorithms

Cited by in corpus (63)

• A Brief Survey of Deep Reinforcement Learning
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Social physics
• Deep Reinforcement Learning: An Overview
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Cooperating with Machines
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Multi-Agent Reinforcement Learning for Markov Routing Games: A New Modeling Paradigm For Dynamic Traffic Assignment
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• A Deep Policy Inference Q-Network for Multi-Agent Systems
• Warmth and competence in human-agent cooperation
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• AI Research Considerations for Human Existential Safety (ARCHES)
• Deep reinforcement learning in World-Earth system models to discover sustainable management strategies
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks
• A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• CERN for AI: A Theoretical Framework for Autonomous Simulation-Based Artificial Intelligence Testing and Alignment
• Facilitating Cooperation in Human-Agent Hybrid Populations through Autonomous Agents
• Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning
• Policy Gradient With Value Function Approximation For Collective Multiagent Planning
• Deep Reinforcement Learning for Green Security Games with Real-Time Information
• Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games
• Reinforcement Learning Under Moral Uncertainty
• Human-Level Performance in No-Press Diplomacy via Equilibrium Search
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Provable Fictitious Play for General Mean-Field Games
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Generosity, selfishness and exploitation as optimal greedy strategies for resource sharing
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Reward Function Design for Crowd Simulation via Reinforcement Learning
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Learning to Deceive in Multi-Agent Hidden Role Games
• Non-cooperative Multi-agent Systems with Exploring Agents
• Joint Attention for Multi-Agent Coordination and Social Learning
• Agent Environment Cycle Games
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning
• Stochastic Market Games
• The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning
• Learning through Probing: a decentralized reinforcement learning architecture for social dilemmas
• Exploitation by asymmetry of information reference in coevolutionary learning in prisoner's dilemma game
• Exploring the Impact of Tunable Agents in Sequential Social Dilemmas
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges
• Situationally Aware Options
• Cooperation Enforcement and Collusion Resistance in Repeated Public Goods Games
• Goal Conflict in Designing an Autonomous Artificial System
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss
• Improved Cooperation by Exploiting a Common Signal