Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

Abstract

Reinforcement learning (RL) algorithms for real-world robotic applications need a data-efficient learning process and the ability to handle complex, unknown dynamical systems. These requirements are handled well by model-based and model-free RL approaches, respectively. In this work, we aim to combine the advantages of these two types of methods in a principled manner. By focusing on time-varying linear-Gaussian policies, we enable a model-based algorithm based on the linear quadratic regulator (LQR) that can be integrated into the model-free framework of path integral policy improvement (PI2). We can further combine our method with guided policy search (GPS) to train arbitrary parameterized policies such as deep neural networks. Our simulation and real-world experiments demonstrate that this method can solve challenging manipulation tasks with comparable or better performance than model-free methods while maintaining the sample efficiency of model-based methods. A video presenting our results is available at https://sites.google.com/site/icml17pilqr

arXiv commentPaper accepted to the International Conference on Machine Learning (ICML) 2017

References in corpus (7)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• End-to-End Training of Deep Visuomotor Policies
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Contact-Rich Manipulation Skills with Guided Policy Search

Cited by in corpus (17)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Exploring Model-based Planning with Policy Networks
• Dynamics-Aware Unsupervised Discovery of Skills
• The Ingredients of Real-World Robotic Reinforcement Learning
• Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly
• Safe Learning-Based Feedback Linearization Tracking Control for Nonlinear System with Event-Triggered Model Update
• RoboNet: Large-Scale Multi-Robot Learning
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives
• Compositional Reinforcement Learning from Logical Specifications
• Adaptive Road Configurations for Improved Autonomous Vehicle-Pedestrian Interactions using Reinforcement Learning
• Habits and goals in synergy: a variational Bayesian framework for behavior
• Stochastic Lipschitz Q-Learning
• Learning Cloth Folding Tasks with Refined Flow Based Spatio-Temporal Graphs
• Visual Reaction: Learning to Play Catch with Your Drone
• On-Policy Model Errors in Reinforcement Learning