Open Problems in Cooperative AI

Abstract

Problems of cooperation--in which agents seek ways to jointly improve their welfare--are ubiquitous and important. They can be found at scales ranging from our daily routines--such as driving on highways, scheduling meetings, and working collaboratively--to our global challenges--such as peace, commerce, and pandemic preparedness. Arguably, the success of the human species is rooted in our ability to cooperate. Since machines powered by artificial intelligence are playing an ever greater role in our lives, it will be important to equip them with the capabilities necessary to cooperate and to foster cooperation. We see an opportunity for the field of artificial intelligence to explicitly focus effort on this class of problems, which we term Cooperative AI. The objective of this research would be to study the many aspects of the problems of cooperation and to innovate in AI to contribute to solving these problems. Central goals include building machine agents with the capabilities needed for cooperation, building tools to foster cooperation in populations of (machine and/or human) agents, and otherwise conducting AI research for insight relevant to problems of cooperation. This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms. However, Cooperative AI is not the union of these existing areas, but rather an independent bet about the productivity of specific kinds of conversations that involve these and other areas. We see opportunity to more explicitly focus on the problem of cooperation, to construct unified theory and vocabulary, and to build bridges with adjacent communities working on cooperation, including in the natural, social, and behavioural sciences.

References in corpus (42)

• Natural Language Processing (almost) from Scratch
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Language Models are Few-Shot Learners
• The Dynamics of Viral Marketing
• Equality of Opportunity in Supervised Learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Dota 2 with Large Scale Deep Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Emergent Tool Use From Multi-Agent Autocurricula
• The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• The Hanabi Challenge: A New Frontier for AI Research
• Cooperating with Machines
• Adversarial Robustness through Local Linearization
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Scalable agent alignment via reward modeling: a research direction
• On the Utility of Learning about Humans for Human-AI Coordination
• Anti-efficient encoding in emergent communication
• Giraffe: Using Deep Reinforcement Learning to Play Chess
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Learning to Understand Goal Specifications by Modelling Reward
• Multi-Agent Common Knowledge Reinforcement Learning
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Emergent Communication through Negotiation
• Towards Playing Full MOBA Games with Deep Reinforcement Learning
• Kickstarting Deep Reinforcement Learning
• Learning to summarize from human feedback
• "Other-Play" for Zero-Shot Coordination
• Occam's razor is insufficient to infer the preferences of irrational agents
• Repeated Inverse Reinforcement Learning
• Observational Learning by Reinforcement Learning
• AI Research Considerations for Human Existential Safety (ARCHES)
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• A Survey on Deep Learning-based Non-Invasive Brain Signals:Recent Advances and New Frontiers
• Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• Stable Opponent Shaping in Differentiable Games
• Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences
• Perfect Prediction Equilibrium
• The 2020s Political Economy of Machine Translation
• Artificial Intelligence & Cooperation

Cited by in corpus (12)

• Unsolved Problems in ML Safety
• Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games
• The Emergence of Wireless MAC Protocols with Multi-Agent Reinforcement Learning
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Bounding the Inefficiency of Route Control in Intelligent Transport Systems
• Reinforcement Learning Under Algorithmic Triage
• How Can Creativity Occur in Multi-Agent Systems?
• Normative Disagreement as a Challenge for Cooperative AI
• Adversarial Attacks in Cooperative AI
• Solving Graph-based Public Good Games with Tree Search and Imitation Learning
• Improving Social Welfare While Preserving Autonomy via a Pareto Mediator
• Targeted Data Acquisition for Evolving Negotiation Agents