Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?

Abstract

The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gait-cycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.

Cited by in corpus (20)

• Learning agile and dynamic motor skills for legged robots
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Multi-expert learning of adaptive legged locomotion
• Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• Learning natural locomotion behaviors for humanoid robots using human knowledge
• Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer
• Learning Whole-body Motor Skills for Humanoids
• On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• Observation Space Matters: Benchmark and Optimization Algorithm
• Emergence of Human-comparable Balancing Behaviors by Deep Reinforcement Learning
• Efficient Reinforcement Learning for Jumping Monopods
• A Survey on Reinforcement Learning Methods in Character Animation
• Efficient Learning of Control Policies for Robust Quadruped Bounding using Pretrained Neural Networks
• Learning to Control Emulated Muscles in Real Robots: Towards Exploiting Bio-Inspired Actuator Morphology