Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Abstract

Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity. However, this is difficult because an imperfect dynamics model can degrade the performance of the learning algorithm, and in sufficiently complex environments, the dynamics model will almost always be imperfect. As a result, a key challenge is to combine model-based approaches with model-free learning in such a way that errors in the model do not degrade performance. We propose stochastic ensemble value expansion (STEVE), a novel model-based technique that addresses this issue. By dynamically interpolating between model rollouts of various horizon lengths for each individual example, STEVE ensures that the model is only utilized when doing so does not introduce significant errors. Our approach outperforms model-free baselines on challenging continuous control benchmarks with an order-of-magnitude increase in sample efficiency, and in contrast to previous model-based approaches, performance does not degrade in complex environments.

Cited by in corpus (53)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Off-Policy Deep Reinforcement Learning without Exploration
• Dream to Control: Learning Behaviors by Latent Imagination
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Mastering Atari with Discrete World Models
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• Trust the Model When It Is Confident: Masked Model-based Actor-Critic
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• On the model-based stochastic value gradient for continuous reinforcement learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?
• Model-based Adversarial Meta-Reinforcement Learning
• Asynchronous Methods for Model-Based Reinforcement Learning
• Gradient-Aware Model-based Policy Search
• Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensembles
• Model-based Lookahead Reinforcement Learning
• GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning
• Value Iteration in Continuous Actions, States and Time
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi-Supervised Object Detection
• Deep Residual Reinforcement Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Selective Dyna-style Planning Under Limited Model Capacity
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Learning Dynamics Models for Model Predictive Agents
• Multi-Path Policy Optimization
• Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving
• Learning Trajectories for Visual-Inertial System Calibration via Model-based Heuristic Deep Reinforcement Learning
• Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
• Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Towards robust and domain agnostic reinforcement learning competitions
• Policy Gradients Incorporating the Future
• Learning Off-Policy with Online Planning
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Planning with Expectation Models for Control
• Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Acting upon Imagination: when to trust imagined trajectories in model based reinforcement learning
• Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation
• Deterministic Value-Policy Gradients
• Discriminator Augmented Model-Based Reinforcement Learning