Consequentialist conditional cooperation in social dilemmas with imperfect information

Abstract

Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction. We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest. However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict. We show that in a large class of games good strategies can be constructed by conditioning one's behavior solely on outcomes (ie. one's past rewards). We call this consequentialist conditional cooperation. We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games. We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.

References in corpus (10)

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Cooperating with Machines
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• A Polynomial-time Nash Equilibrium Algorithm for Repeated Stochastic Games
• Emergent Communication in a Multi-Modal, Multi-Step Referential Game

Cited by in corpus (9)

• Warmth and competence in human-agent cooperation
• Backplay: "Man muss immer umkehren"
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Birds of a Feather Flock Together: A Close Look at Cooperation Emergence via Multi-Agent RL
• Learning in two-player games between transparent opponents
• Improved Cooperation by Exploiting a Common Signal
• Normative Disagreement as a Challenge for Cooperative AI