Surprising Negative Results for Generative Adversarial Tree Search

Abstract

While many recent advances in deep reinforcement learning (RL) rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment model. In this work, we provide an extensive study on the design of deep generative models for RL environments and propose a sample efficient and robust method to learn the model of Atari environments. We deploy this model and propose generative adversarial tree search (GATS) a deep RL algorithm that learns the environment model and implements Monte Carlo tree search (MCTS) on the learned model for planning. While MCTS on the learned model is computationally expensive, similar to AlphaGo, GATS follows depth limited MCTS. GATS employs deep Q network (DQN) and learns a Q-function to assign values to the leaves of the tree in MCTS. We theoretical analyze GATS vis-a-vis the bias-variance trade-off and show GATS is able to mitigate the worst-case error in the Q-estimate. While we were expecting GATS to enjoy a better sample complexity and faster converges to better policies, surprisingly, GATS fails to outperform DQN. We provide a study on which we show why depth limited MCTS fails to perform desirably.

References in corpus (12)

• Conditional Generative Adversarial Nets
• Spectral Normalization for Generative Adversarial Networks
• Trust Region Policy Optimization
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Noisy Networks for Exploration
• Count-Based Exploration with Neural Density Models
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• A Bayesian Sampling Approach to Exploration in Reinforcement Learning
• REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces
• Reinforcement Learning in Rich-Observation MDPs using Spectral Methods

Cited by in corpus (8)

• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Combining Q-Learning and Search with Amortized Value Estimates
• On the role of planning in model-based deep reinforcement learning
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Structured agents for physical construction
• Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning