Model-Based Opponent Modeling

Abstract

When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponents. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponents. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in a variety of tasks, respectively with different types of opponents, i.e., fixed policy, naïve learner, and reasoning learner.

arXiv comment21 pages, 10 figures

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Model-Ensemble Trust-Region Policy Optimization
• On-line Policy Improvement using Monte-Carlo Search
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Learning Policy Representations in Multiagent Systems
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents
• Multi-Agent Reinforcement Learning with Multi-Step Generative Models
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems