Neural Recursive Belief States in Multi-Agent Reinforcement Learning

Abstract

In multi-agent reinforcement learning, the problem of learning to act is particularly difficult because the policies of co-players may be heavily conditioned on information only observed by them. On the other hand, humans readily form beliefs about the knowledge possessed by their peers and leverage beliefs to inform decision-making. Such abilities underlie individual success in a wide range of Markov games, from bluffing in Poker to conditional cooperation in the Prisoner's Dilemma, to convention-building in Bridge. Classical methods are usually not applicable to complex domains due to the intractable nature of hierarchical beliefs (i.e. beliefs of other agents' beliefs). We propose a scalable method to approximate these belief structures using recursive deep generative models, and to use the belief models to obtain representations useful to acting in complex tasks. Our agents trained with belief models outperform model-free baselines with equivalent representational capacity using common training paradigms. We also show that higher-order belief models outperform agents with lower-order models.

References in corpus (14)

• Auto-Encoding Variational Bayes
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• MADE: Masked Autoencoder for Distribution Estimation
• The Hanabi Challenge: A New Frontier for AI Research
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
• Monte Carlo Sampling Methods for Approximating Interactive POMDPs
• Machine Theory of Mind
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Meta reinforcement learning as task inference
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Neural Predictive Belief Representations
• Autoregressive Energy Machines
• On The Identifiability of Mixture Models from Grouped Samples