Maintaining cooperation in complex social dilemmas using deep reinforcement learning

Abstract

Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment.

References in corpus (9)

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Cooperating with Machines
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• Emergent Communication in a Multi-Modal, Multi-Step Referential Game

Cited by in corpus (30)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Social physics
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• "Other-Play" for Zero-Shot Coordination
• Backplay: "Man muss immer umkehren"
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Consequentialist conditional cooperation in social dilemmas with imperfect information
• Finding Friend and Foe in Multi-Agent Games
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Learning to Incentivize Other Learning Agents
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Model-Based Opponent Modeling
• Human-Level Performance in No-Press Diplomacy via Equilibrium Search
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Robust Multi-agent Counterfactual Prediction
• Evaluating and Rewarding Teamwork Using Cooperative Game Abstractions
• Learning through Probing: a decentralized reinforcement learning architecture for social dilemmas
• Learning in two-player games between transparent opponents
• Stochastic Market Games
• Statistical discrimination in learning agents
• Learn Task First or Learn Human Partner First: A Hierarchical Task Decomposition Method for Human-Robot Cooperation
• Individual specialization in multi-task environments with multiagent reinforcement learners
• Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss
• Negotiating Team Formation Using Deep Reinforcement Learning
• Loss aversion fosters coordination among independent reinforcement learners
• Searching with Opponent-Awareness
• Normative Disagreement as a Challenge for Cooperative AI