Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

Abstract

Humans are capable of attributing latent mental contents such as beliefs or intentions to others. The social skill is critical in daily life for reasoning about the potential consequences of others' behaviors so as to plan ahead. It is known that humans use such reasoning ability recursively by considering what others believe about their own beliefs. In this paper, we start from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policies, to which each agent finds the best response and then improve their own policies. We develop decentralized-training-decentralized-execution algorithms, namely PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenarios when there exists one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

ICLR 2019

Cited by in corpus (26)

• Game-Theoretic Multiagent Reinforcement Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Settling the Variance of Multi-Agent Policy Gradients
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Multi-Agent Constrained Policy Optimisation
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Networked Multi-Agent Reinforcement Learning with Emergent Communication
• Model-Based Opponent Modeling
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Learning to Communicate Implicitly By Actions
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World
• A Regularized Opponent Model with Maximum Entropy Objective
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• Shaping Advice in Deep Multi-Agent Reinforcement Learning
• Bi-level Actor-Critic for Multi-agent Coordination
• Learning Complex Multi-Agent Policies in Presence of an Adversary
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• L2E: Learning to Exploit Your Opponent
• Improving Dialog Systems for Negotiation with Personality Modeling
• Agent Probing Interaction Policies