A Deep Policy Inference Q-Network for Multi-Agent Systems

Abstract

We present DPIQN, a deep policy inference Q-network that targets multi-agent systems composed of controllable agents, collaborators, and opponents that interact with each other. We focus on one challenging issue in such systems---modeling agents with varying strategies---and propose to employ "policy features" learned from raw observations (e.g., raw images) of collaborators and opponents by inferring their policies. DPIQN incorporates the learned policy features as a hidden vector into its own deep Q-network (DQN), such that it is able to predict better Q values for the controllable agents than the state-of-the-art deep reinforcement learning models. We further propose an enhanced version of DPIQN, called deep recurrent policy inference Q-network (DRPIQN), for handling partial observability. Both DPIQN and DRPIQN are trained by an adaptive training procedure, which adjusts the network's attention to learn the policy features and its own Q-values at different phases of the training process. We present a comprehensive analysis of DPIQN and DRPIQN, and highlight their effectiveness and generalizability in various multi-agent settings. Our models are evaluated in a classic soccer game involving both competitive and collaborative scenarios. Experimental results performed on 1 vs. 1 and 2 vs. 2 games show that DPIQN and DRPIQN demonstrate superior performance to the baseline DQN and deep recurrent Q-network (DRQN) models. We also explore scenarios in which collaborators or opponents dynamically change their policies, and show that DPIQN and DRPIQN do lead to better overall performance in terms of stability and mean scores.

References in corpus (7)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Learning to Navigate in Complex Environments
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Loss is its own Reward: Self-Supervision for Reinforcement Learning

Cited by in corpus (6)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• Efficient Use of heuristics for accelerating XCS-based Policy Learning in Markov Games