Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
arXiv:1812.00045
Abstract
Deep reinforcement learning (DRL) has achieved great successes in recent years with the help of novel methods and higher compute power. However, there are still several challenges to be addressed such as convergence to locally optimal policies and long training times. In this paper, firstly, we augment Asynchronous Advantage Actor-Critic (A3C) method with a novel self-supervised auxiliary task, i.e. \emph{Terminal Prediction}, measuring temporal closeness to terminal states, namely A3C-TP. Secondly, we propose a new framework where planning algorithms such as Monte Carlo tree search or other sources of (simulated) demonstrators can be integrated to asynchronous distributed DRL methods. Compared to vanilla A3C, our proposed methods both learn faster and converge to better policies on a two-player mini version of the Pommerman game.
arXiv comment9 pages, 6 figures, To appear at AAAI-19 Workshop on Reinforcement Learning in Games
References in corpus (10)
- A Brief Survey of Deep Reinforcement Learning
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Deep Reinforcement Learning: An Overview
- Deep reinforcement learning from human preferences
- Thinking Fast and Slow with Deep Learning and Tree Search
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Learning Montezuma's Revenge from a Single Demonstration
- Backplay: "Man muss immer umkehren"
- Surprising Negative Results for Generative Adversarial Tree Search