RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion
arXiv:2305.17842 · doi:10.1109/LRA.2023.3307008
Abstract
This paper presents a control framework that combines model-based optimal control and reinforcement learning (RL) to achieve versatile and robust legged locomotion. Our approach enhances the RL training process by incorporating on-demand reference motions generated through finite-horizon optimal control, covering a broad range of velocities and gaits. These reference motions serve as targets for the RL policy to imitate, leading to the development of robust control policies that can be learned with reliability. Furthermore, by utilizing realistic simulation data that captures whole-body dynamics, RL effectively overcomes the inherent limitations in reference motions imposed by modeling simplifications. We validate the robustness and controllability of the RL training process within our framework through a series of experiments. In these experiments, our method showcases its capability to generalize reference motions and effectively handle more complex locomotion tasks that may pose challenges for the simplified model, thanks to RL's flexibility. Additionally, our framework effortlessly supports the training of control policies for robots with diverse dimensions, eliminating the necessity for robot-specific adjustments in the reward function and hyperparameters.
The paper has been accepted for publication in IEEE Robotics and Automation Letters (RA-L). You can find the copyright information on the front page of the paper. The supplementary video is available in https://www.youtube.com/watch?v=qPttVfzGS84
References in corpus (13)
- Learning agile and dynamic motor skills for legged robots
- Learning Quadrupedal Locomotion over Challenging Terrain
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Learning robust perceptive locomotion for quadrupedal robots in the wild
- Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control
- Motion Planning for Quadrupedal Locomotion: Coupled Planning, Terrain Mapping and Whole-Body Control
- Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
- Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller
- Kinodynamic Motion Planning for Multi-Legged Robot Jumping via Mixed-Integer Convex Program
- Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior
- Nonlinear Model Predictive Control for Quadrupedal Locomotion Using Second-Order Sensitivity Analysis
- Optimization-Based Control for Dynamic Legged Robots
- Reinforcement Learning for Legged Robots: Motion Imitation from Model-Based Optimal Control