Modeling Others using Oneself in Multi-Agent Reinforcement Learning

Abstract

We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility. The reward function depends on the hidden state (or goal) of both agents, so the agents must infer the other players' hidden goals from their observed behavior in order to solve the tasks. We propose a new approach for learning in these domains: Self Other-Modeling (SOM), in which an agent uses its own policy to predict the other agent's actions and update its belief of their hidden state in an online manner. We evaluate this approach on three different tasks and show that the agents are able to learn better policies using their estimate of the other players' hidden states, in both cooperative and adversarial settings.

arXiv comment10 pages, 16 figures, submitted to ICML 2018

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Cooperative Inverse Reinforcement Learning
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Learning Multiagent Communication with Backpropagation
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks
• Inverse Reward Design
• MazeBase: A Sandbox for Learning from Games
• It Takes Two to Tango: Towards Theory of AI's Mind

Cited by in corpus (29)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• OpenSpiel: A Framework for Reinforcement Learning in Games
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
• Concurrent Meta Reinforcement Learning
• Learning Latent Representations to Influence Multi-Agent Interaction
• Learning Latent Representations to Co-Adapt to Humans
• Entropy Regularization for Mean Field Games with Learning
• Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games
• Learning to Communicate Implicitly By Actions
• A Regularized Opponent Model with Maximum Entropy Objective
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• Joint Attention for Multi-Agent Coordination and Social Learning
• Blind interactive learning of modulation schemes: Multi-agent cooperation without co-design
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Competitive Policy Optimization
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Towards Empathic Deep Q-Learning
• Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation
• Learning to Play Sequential Games versus Unknown Opponents
• Multi-Turn Beam Search for Neural Dialogue Modeling
• Ubiquitous Distributed Deep Reinforcement Learning at the Edge: Analyzing Byzantine Agents in Discrete Action Spaces
• RL agents Implicitly Learning Human Preferences
• To be a fast adaptive learner: using game history to defeat opponents
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• A Bayesian Approach to Identifying Representational Errors
• Directionality Reinforcement Learning to Operate Multi-Agent System without Communication