Opponent Modeling in Deep Reinforcement Learning

Abstract

Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change. Most previous work focuses on developing probabilistic models or parameterized strategies for specific applications. Inspired by the recent success of deep reinforcement learning, we present neural-based models that jointly learn a policy and the behavior of opponents. Instead of explicitly predicting the opponent's action, we encode observation of the opponents into a deep Q-Network (DQN); however, we retain explicit modeling (if desired) using multitasking. By using a Mixture-of-Experts architecture, our model automatically discovers different strategy patterns of opponents without extra supervision. We evaluate our models on a simulated soccer game and a popular trivia game, showing superior performance over DQN and its variants.

In ICML 2016

References in corpus (4)

• Deep Recurrent Q-Learning for Partially Observable MDPs
• Bayes' Bluff: Opponent Modelling in Poker
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks
• Learning Deep Neural Network Policies with Continuous Memory States

Cited by in corpus (13)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG
• Learning Models of Individual Behavior in Chess
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
• Learning Complex Multi-Agent Policies in Presence of an Adversary
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics