Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Abstract

Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).

arXiv commentNIPS 2018, video and code available at https://sites.google.com/view/drl-in-a-handful-of-trials/

References in corpus (10)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Searching for Activation Functions
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Deep Exploration via Bootstrapped DQN
• Model-Ensemble Trust-Region Policy Optimization
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning

Cited by in corpus (54)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• E(n) Equivariant Graph Neural Networks
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning to Walk in the Real World with Minimal Human Effort
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Exploring Model-based Planning with Policy Networks
• Planning to Explore via Self-Supervised World Models
• Residual Policy Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
• Self-Supervised Exploration via Disagreement
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Gradient-Aware Model-based Policy Search
• Reset-Free Lifelong Learning with Skill-Space Planning
• Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Prediction
• DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Adaptive Online Planning for Continual Lifelong Learning
• Safety Guarantees for Planning Based on Iterative Gaussian Processes
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Low Emission Building Control with Zero-Shot Reinforcement Learning
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• Inference of Affordances and Active Motor Control in Simulated Agents
• Model Imitation for Model-Based Reinforcement Learning
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Model-Based Reinforcement Learning for Type 1Diabetes Blood Glucose Control
• Data-efficient Learning of Morphology and Controller for a Microrobot
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Learning Dynamical Systems from Noisy Sensor Measurements using Multiple Shooting
• Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• The Impact of Data on the Stability of Learning-Based Control- Extended Version
• Potential Field Guided Actor-Critic Reinforcement Learning
• GEM: Group Enhanced Model for Learning Dynamical Control Systems
• Minimax Model Learning
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• Acting upon Imagination: when to trust imagined trajectories in model based reinforcement learning