Benchmarking Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes closed-sourced or not reproducible. Accordingly, it is an open question how these various existing MBRL algorithms perform relative to each other. To facilitate research in MBRL, in this paper we gather a wide collection of MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to maximally facilitate future research on MBRL, we open-source our benchmark in http://www.cs.toronto.edu/~tingwuwang/mbrl.html.

8 main pages, 8 figures; 14 appendix pages, 25 figures

References in corpus (14)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Emergence of Locomotion Behaviours in Rich Environments
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• DeepMind Control Suite
• Deep Reinforcement Learning that Matters
• Model-Ensemble Trust-Region Policy Optimization
• Model-Based Bayesian Exploration
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Cited by in corpus (88)

• Model-Based Reinforcement Learning for Atari
• MOPO: Model-based Offline Policy Optimization
• MOReL : Model-Based Offline Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Machine learning for industrial sensing and control: A survey and practical perspective
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Deep Model Predictive Variable Impedance Control
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Behaviour Suite for Reinforcement Learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Mastering Atari with Discrete World Models
• Rethinking Closed-loop Training for Autonomous Driving
• What Matters for Adversarial Imitation Learning?
• On the model-based stochastic value gradient for continuous reinforcement learning
• Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
• Combining Q-Learning and Search with Amortized Value Estimates
• On the role of planning in model-based deep reinforcement learning
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Asynchronous Methods for Model-Based Reinforcement Learning
• UniCon: Universal Neural Controller For Physics-based Character Motion
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Convergence results for an averaged LQR problem with applications to reinforcement learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Ready Policy One: World Building Through Active Learning
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Policy-Aware Model Learning for Policy Gradient Methods
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Is Plug-in Solver Sample-Efficient for Feature-based Reinforcement Learning?
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning
• Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Minimax Sample Complexity for Turn-based Stochastic Game
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Model Imitation for Model-Based Reinforcement Learning
• Procedural Generalization by Planning with Self-Supervised World Models
• Continuous Control for Searching and Planning with a Learned Model
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• Planning from Pixels using Inverse Dynamics Models
• On Effective Scheduling of Model-based Reinforcement Learning
• Non-Parametric Neuro-Adaptive Control Subject to Task Specifications
• MobILE: Model-Based Imitation Learning From Observation Alone
• A Tensor Network Approach to Finite Markov Decision Processes
• Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data
• Safe Exploration by Solving Early Terminated MDP
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
• MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Affordance as general value function: A computational model
• Baconian: A Unified Open-source Framework for Model-Based Reinforcement Learning
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?
• Policy Gradients Incorporating the Future
• A survey of benchmarking frameworks for reinforcement learning
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Model Embedding Model-Based Reinforcement Learning
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Optimal Control-Based Baseline for Guided Exploration in Policy Gradient Methods
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Improving gearshift controllers for electric vehicles with reinforcement learning
• Discriminator Augmented Model-Based Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• Domain Knowledge Integration By Gradient Matching For Sample-Efficient Reinforcement Learning