Data Efficient Reinforcement Learning for Legged Robots

Abstract

We present a model-based framework for robot locomotion that achieves walking based on only 4.5 minutes (45,000 control steps) of data collected on a quadruped robot. To accurately model the robot's dynamics over a long horizon, we introduce a loss function that tracks the model's prediction over multiple timesteps. We adapt model predictive control to account for planning latency, which allows the learned model to be used for real time control. Additionally, to ensure safe exploration during model learning, we embed prior knowledge of leg trajectories into the action space. The resulting system achieves fast and robust locomotion. Unlike model-free methods, which optimize for a particular task, our planner can use the same learned dynamics for various tasks, simply by changing the reward function. To the best of our knowledge, our approach is more than an order of magnitude more sample efficient than current model-free methods.

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning agile and dynamic motor skills for legged robots
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• Learning Latent Dynamics for Planning from Pixels
• Whole-Body Nonlinear Model Predictive Control Through Contacts for Quadrupeds
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Policies Modulating Trajectory Generators
• Learning to Walk via Deep Reinforcement Learning
• Self-Correcting Models for Model-Based Reinforcement Learning

Cited by in corpus (2)

• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Quadruped Locomotion on Non-Rigid Terrain using Reinforcement Learning