Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
arXiv:1809.01906
Abstract
This paper proposes a new optimization objective for value-based deep reinforcement learning. We extend conventional Deep Q-Networks (DQNs) by adding a model-learning component yielding a transcoder network. The prediction errors for the model are included in the basic DQN loss as additional regularizers. This augmented objective leads to a richer training signal that provides feedback at every time step. Moreover, because learning an environment model shares a common structure with the RL problem, we hypothesize that the resulting objective improves both sample efficiency and performance. We empirically confirm our hypothesis on a range of 20 games from the Atari benchmark attaining superior results over vanilla DQN without model-based regularization.
arXiv commentPresented at the NIPS Deep Reinforcement Learning Workshop, Montreal, Canada, 2018
References in corpus (12)
- Continuous control with deep reinforcement learning
- Action-Conditional Video Prediction using Deep Networks in Atari Games
- Learning Continuous Control Policies by Stochastic Value Gradients
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Imagination-Augmented Agents for Deep Reinforcement Learning
- Temporal Difference Models: Model-Free Deep RL for Model-Based Control
- Recurrent Environment Simulators
- Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
- Learning and Querying Fast Generative Models for Reinforcement Learning
- Value Prediction Network
- A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games
- Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft