Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning

Abstract

Substantial advancements to model-based reinforcement learning algorithms have been impeded by the model-bias induced by the collected data, which generally hurts performance. Meanwhile, their inherent sample efficiency warrants utility for most robot applications, limiting potential damage to the robot and its environment during training. Inspired by information theoretic model predictive control and advances in deep reinforcement learning, we introduce Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free method that combines model predictive rollouts with policy optimization as to mitigate model bias. MoPAC leverages optimal trajectories to guide policy learning, but explores via its model-free method, allowing the algorithm to learn more expressive dynamics models. This combination guarantees optimal skill learning up to an approximation error and reduces necessary physical interaction with the environment, making it suitable for real-robot training. We provide extensive results showcasing how our proposed method generally outperforms current state-of-the-art and conclude by evaluating MoPAC for learning on a physical robotic hand performing valve rotation and finger gaiting--a task that requires grasping, manipulation, and then regrasping of an object.

arXiv commentIEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 2021

References in corpus (16)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols
• Model-Ensemble Trust-Region Policy Optimization
• Differentiable MPC for End-to-end Planning and Control
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Exploring Model-based Planning with Policy Networks
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Model-Augmented Actor-Critic: Backpropagating through Paths
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• Deep Value Model Predictive Control