Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

Abstract

In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation learning and multiagent deep reinforcement learning, we propose two architectures to perform agent modeling: the first one based on parameter sharing, and the second one based on agent policy features. Both architectures aim to learn other agents' policies as auxiliary tasks, besides the standard actor (policy) and critic (values). We performed experiments in both cooperative and competitive domains. The former is a problem of coordinated multiagent object transportation and the latter is a two-player mini version of the Pommerman game. Our results show that the proposed architectures stabilize learning and outperform the standard A3C architecture when learning a best response in terms of expected rewards.

AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE'19)

Cited by in corpus (10)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• On the Importance of Opponent Modeling in Auction Markets
• Multi-robot Cooperative Object Transportation using Decentralized Deep Reinforcement Learning
• Learning to Classify and Imitate Trading Agents in Continuous Double Auction Markets
• Work in Progress: Temporally Extended Auxiliary Tasks
• RL agents Implicitly Learning Human Preferences