Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning

Abstract

Though limited in real-world decision making, most multi-agent reinforcement learning (MARL) models assume perfectly rational agents -- a property hardly met due to individual's cognitive limitation and/or the tractability of the decision problem. In this paper, we introduce generalized recursive reasoning (GR2) as a novel framework to model agents with different \emph{hierarchical} levels of rationality; our framework enables agents to exhibit varying levels of "thinking" ability thereby allowing higher-level agents to best respond to various less sophisticated learners. We contribute both theoretically and empirically. On the theory side, we devise the hierarchical framework of GR2 through probabilistic graphical models and prove the existence of a perfect Bayesian equilibrium. Within the GR2, we propose a practical actor-critic solver, and demonstrate its convergent property to a stationary point in two-player games through Lyapunov analysis. On the empirical side, we validate our findings on a variety of MARL benchmarks. Precisely, we first illustrate the hierarchical thinking process on the Keynes Beauty Contest, and then demonstrate significant improvements compared to state-of-the-art opponent modeling baselines on the normal-form games and the cooperative navigation benchmark.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Density estimation using Real NVP
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Opponent Modeling in Deep Reinforcement Learning
• Machine Theory of Mind
• Multiagent Soft Q-Learning
• Bridging Level-K to Nash Equilibrium

Cited by in corpus (3)

• Compositional ADAM: An Adaptive Compositional Solver
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization