Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Abstract

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

arXiv commentAppeared in ICLR 2018; typos corrected

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Hindsight Experience Replay
• A Distributional Perspective on Reinforcement Learning
• Learning Parameterized Skills
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Goal-Driven Dynamics Learning via Bayesian Optimization

Cited by in corpus (49)

• Visual Reinforcement Learning with Imagined Goals
• Exploration by Random Network Distillation
• Differentiable MPC for End-to-end Planning and Control
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning Plannable Representations with Causal InfoGAN
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Mapping State Space using Landmarks for Universal Goal Reaching
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Learning to Reach Goals via Iterated Supervised Learning
• Deep reinforcement learning in World-Earth system models to discover sustainable management strategies
• Is Q-learning Provably Efficient?
• Deep Reinforcement and InfoMax Learning
• Learning Latent Plans from Play
• World Model as a Graph: Learning Latent Landmarks for Planning
• Sparse Graphical Memory for Robust Planning
• Training Agents using Upside-Down Reinforcement Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Time Reversal as Self-Supervision
• The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
• Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems
• Model-based Lookahead Reinforcement Learning
• Active Feature Acquisition with Generative Surrogate Models
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• The Differentiable Cross-Entropy Method
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Regularizing Trajectory Optimization with Denoising Autoencoders
• Deep Quality-Value (DQV) Learning
• The Laplacian in RL: Learning Representations with Efficient Approximations
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery
• Outcome-Driven Reinforcement Learning via Variational Inference
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• Hierarchical Robot Navigation in Novel Environments using Rough 2-D Maps
• Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Connecting Context-specific Adaptation in Humans to Meta-learning
• Chrome Dino Run using Reinforcement Learning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
• Understanding the World Through Action
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• A Hybrid PAC Reinforcement Learning Algorithm