Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
arXiv:1906.02138
Abstract
This paper investigates the use of intrinsic reward to guide exploration in multi-agent reinforcement learning. We discuss the challenges in applying intrinsic reward to multiple collaborative agents and demonstrate how unreliable reward can prevent decentralized agents from learning the optimal policy. We address this problem with a novel framework, Independent Centrally-assisted Q-learning (ICQL), in which decentralized agents share control and an experience replay buffer with a centralized agent. Only the centralized agent is intrinsically rewarded, but the decentralized agents still benefit from improved exploration, without the distraction of unreliable incentives.
arXiv commentAccepted to the 2nd Exploration in Reinforcement Learning Workshop at the International Conference on Machine Learning 2019
References in corpus (6)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Counterfactual Multi-Agent Policy Gradients
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
Cited by in corpus (8)
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
- Deep Coordination Graphs
- Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
- Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
- REMAX: Relational Representation for Multi-Agent Exploration
- Cooperative and Competitive Biases for Multi-Agent Reinforcement Learning