Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
arXiv:1904.05759
Abstract
Safe reinforcement learning has many variants and it is still an open research problem. Here, we focus on how to use action guidance by means of a non-expert demonstrator to avoid catastrophic events in a domain with sparse, delayed, and deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. This domain is very challenging for reinforcement learning (RL) --- past work has shown that model-free RL algorithms fail to achieve significant learning. In this paper, we shed light into the reasons behind this failure by exemplifying and analyzing the high rate of catastrophic events (i.e., suicides) that happen under random exploration in this domain. While model-free random exploration is typically futile, we propose a new framework where even a non-expert simulated demonstrator, e.g., planning algorithms such as Monte Carlo tree search with small number of rollouts, can be integrated to asynchronous distributed deep reinforcement learning methods. Compared to vanilla deep RL algorithms, our proposed methods both learn faster and converge to better policies on a two-player mini version of the Pommerman game.
arXiv commentAdaptive Learning Agents (ALA) Workshop at AAMAS 2019. arXiv admin note: substantial text overlap with arXiv:1812.00045
References in corpus (10)
- A Brief Survey of Deep Reinforcement Learning
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- Deep Reinforcement Learning: An Overview
- Go-Explore: a New Approach for Hard-Exploration Problems
- AI Safety Gridworlds
- Safe Policy Improvement by Minimizing Robust Baseline Regret
- Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
- Deep Multi-Agent Reinforcement Learning with Relevance Graphs
- Backplay: "Man muss immer umkehren"
- Real-time tree search with pessimistic scenarios