Generative Temporal Difference Learning for Infinite-Horizon Prediction

Abstract

We introduce the $\gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with $\gamma$-models leads to generalizations of the procedures central to model-based control, including the model rollout and model-based value estimation. The $\gamma$-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the $\gamma$-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.

arXiv commentNeurIPS 2020. Project page at: https://gammamodels.github.io/

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Variational Inference with Normalizing Flows
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Learning Continuous Control Policies by Stochastic Value Gradients
• Differentiable MPC for End-to-end Planning and Control
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• The Predictron: End-To-End Learning and Planning
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Combating the Compounding-Error Problem with a Multi-step Model
• Universal Successor Representations for Transfer Reinforcement Learning