Learning Dynamics Models for Model Predictive Agents
arXiv:2109.14311
Abstract
Model-Based Reinforcement Learning involves learning a \textit{dynamics model} from data, and then using this model to optimise behaviour, most often with an online \textit{planner}. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model -- the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.
References in corpus (12)
- Continuous control with deep reinforcement learning
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- DeepMind Control Suite
- Learning Latent Dynamics for Planning from Pixels
- Model-Ensemble Trust-Region Policy Optimization
- MOPO: Model-based Offline Policy Optimization
- Maximum a Posteriori Policy Optimisation
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- MOReL : Model-Based Offline Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- A Differentiable Newton Euler Algorithm for Multi-body Model Learning