On the model-based stochastic value gradient for continuous reinforcement learning

Abstract

For over a decade, model-based reinforcement learning has been seen as a way to leverage control-based domain knowledge to improve the sample-efficiency of reinforcement learning agents. While model-based agents are conceptually appealing, their policies tend to lag behind those of model-free agents in terms of final reward, especially in non-trivial environments. In response, researchers have proposed model-based agents with increasingly complex components, from ensembles of probabilistic dynamics models, to heuristics for mitigating model error. In a reversal of this trend, we show that simple model-based agents can be derived from existing ideas that not only match, but outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. We find that a model-free soft value estimate for policy evaluation and a model-based stochastic value gradient for policy improvement is an effective combination, achieving state-of-the-art results on a high-dimensional humanoid control task, which most model-based agents are unable to solve. Our findings suggest that model-based policy evaluation deserves closer attention.

arXiv commentL4DC 2021

References in corpus (40)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The NumPy array: a structure for efficient numerical computation
• Continuous control with deep reinforcement learning
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Dota 2 with Large Scale Deep Reinforcement Learning
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Learning Continuous Control Policies by Stochastic Value Gradients
• Continuous Deep Q-Learning with Model-based Acceleration
• Safe Exploration in Continuous Action Spaces
• Off-Policy Deep Reinforcement Learning without Exploration
• Model-Ensemble Trust-Region Policy Optimization
• Behavior Regularized Offline Reinforcement Learning
• Benchmarking Model-Based Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Differentiable MPC for End-to-end Planning and Control
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• When to Trust Your Model: Model-Based Policy Optimization
• When to use parametric models in reinforcement learning?
• Exploring Model-based Planning with Policy Networks
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Taming the Noise in Reinforcement Learning via Soft Updates
• Planning to Explore via Self-Supervised World Models
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Path Integral Networks: End-to-End Differentiable Optimal Control
• Value constrained model-free continuous control
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• Self-Supervised Exploration via Disagreement
• On the role of planning in model-based deep reinforcement learning
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Local Search for Policy Iteration in Continuous Control
• Infinite-Horizon Differentiable Model Predictive Control
• Iterative Amortized Policy Optimization

Cited by in corpus (6)

• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• Evaluating model-based planning and planner amortization for continuous control
• Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models