Learning agile and dynamic motor skills for legged robots

Abstract

Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is complicated and expensive. In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes. The approach is applied to the ANYmal robot, a sophisticated medium-dog-sized quadrupedal system. Using policies trained in simulation, the quadrupedal machine achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.

References in corpus (9)

• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
• Emergence of Locomotion Behaviours in Rich Environments
• Control of a Quadrotor with Reinforcement Learning
• Learning Symmetric and Low-energy Locomotion
• Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?
• An Efficient Optimal Planning and Control Framework For Quadrupedal Locomotion
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Cited by in corpus (205)

• On the Opportunities and Risks of Foundation Models
• Learning Quadrupedal Locomotion over Challenging Terrain
• Learning robust perceptive locomotion for quadrupedal robots in the wild
• Solving Rubik's Cube with a Robot Hand
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
• Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms
• Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control
• A Survey of Machine Learning for Computer Architecture and Systems
• A soft robot that adapts to environments through shape change
• Multi-expert learning of adaptive legged locomotion
• Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
• Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
• Representation-Free Model Predictive Control for Dynamic Motions in Quadrupeds
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control
• Cat-like Jumping and Landing of Legged Robots in Low-gravity Using Deep Reinforcement Learning
• Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots
• DTC: Deep Tracking Control
• Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning
• Learning to Walk in the Real World with Minimal Human Effort
• Driving in Dense Traffic with Model-Free Reinforcement Learning
• Towards Expedited Impedance Tuning of a Robotic Prosthesis for Personalized Gait Assistance by Reinforcement Learning Control
• Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller
• Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Data-Driven Multi-Objective Controller Optimization for a Magnetically-Levitated Nanopositioning System
• Learning natural locomotion behaviors for humanoid robots using human knowledge
• MPC-Net: A First Principles Guided Policy Search
• Automated shapeshifting for function recovery in damaged robots
• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger
• Reinforcement Learning Applications
• RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion
• Reinforcement Learning of CPG-regulated Locomotion Controller for a Soft Snake Robot
• Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics
• Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning
• Imitation Learning from MPC for Quadrupedal Multi-Gait Control
• HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Learning Bipedal Walking for Humanoids with Current Feedback
• Testing of Deep Reinforcement Learning Agents with Surrogate Models
• Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running
• Grasp Multiple Objects with One Hand
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
• Robust Quadruped Jumping via Deep Reinforcement Learning
• A general locomotion control framework for multi-legged locomotors
• Meta Reinforcement Learning for Optimal Design of Legged Robots
• Data Efficient Reinforcement Learning for Legged Robots
• Contact-Implicit Model Predictive Control: Controlling Diverse Quadruped Motions Without Pre-Planned Contact Modes or Trajectories
• Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• Self-propulsion via slipping: frictional swimming in multi-legged locomotors
• Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer
• TacSL: A Library for Visuotactile Sensor Simulation and Learning
• Distributed Model Predictive Safety Certification for Learning-based Control
• Deep Value Model Predictive Control
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion
• Sim-to-Real of Soft Robots with Learned Residual Physics
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer
• Learning coordinated badminton skills for legged manipulators
• Generalization in Transfer Learning
• The need for and feasibility of alternative ground robots to traverse sandy and rocky extraterrestrial terrain
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante
• Omni-Roach: A legged robot capable of traversing multiple types of large obstacles and self-righting
• RMA: Rapid Motor Adaptation for Legged Robots
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• A Legged Soft Robot Platform for Dynamic Locomotion
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• NimbRo-OP2X: Affordable Adult-sized 3D-printed Open-Source Humanoid Robot for Research
• Trojan Playground: A Reinforcement Learning Framework for Hardware Trojan Insertion and Detection
• Observation Space Matters: Benchmark and Optimization Algorithm
• Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functions
• Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap
• Real-Time Model Calibration with Deep Reinforcement Learning
• Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
• Learning Fast Adaptation with Meta Strategy Optimization
• Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Learning Physical Constraints with Neural Projections
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Learning Quadrotor Control From Visual Features Using Differentiable Simulation
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• FR-Net: Learning Robust Quadrupedal Fall Recovery on Challenging Terrains through Mass-Contact Prediction
• Traversing the Reality Gap via Simulator Tuning
• Smooth Exploration for Robotic Reinforcement Learning
• Learning Agile Swimming: An End-to-End Approach without CPGs
• Fast and Efficient Locomotion via Learned Gait Transitions
• Zero-Shot Terrain Generalization for Visual Locomotion Policies
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Coordinated Crawling via Reinforcement Learning
• End-to-End Reinforcement Learning for Torque Based Variable Height Hopping
• $H_{\infty }$ Model-free Reinforcement Learning with Robust Stability Guarantee
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning
• MEVIUS: A Quadruped Robot Easily Constructed through E-Commerce with Sheet Metal Welding and Machining
• Word2vec to behavior: morphology facilitates the grounding of language in machines
• Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion
• Robust Quadrupedal Locomotion on Sloped Terrains: A Linear Policy Approach
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
• Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
• A Nearly Optimal Chattering Reduction Method of Sliding Mode Control With an Application to a Two-wheeled Mobile Robot
• Proprioceptive External Torque Learning for Floating Base Robot and its Applications to Humanoid Locomotion
• Informed Guided Rapidly-Exploring Random Trees*-Connect for Path Planning of Walking Robots
• Learning Quadrupedal Locomotion for a Heavy Hydraulic Robot Using an Actuator Model
• Safe Value Functions
• Non-Gaited Legged Locomotion with Monte-Carlo Tree Search and Supervised Learning
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• Teach Biped Robots to Walk via Gait Principles and Reinforcement Learning with Adversarial Critics
• Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning
• Learning Agile Locomotion via Adversarial Training
• Relevance-guided Unsupervised Discovery of Abilities with Quality-Diversity Algorithms
• Comparing Semi-Parametric Model Learning Algorithms for Dynamic Model Estimation in Robotics
• Mixed-Reality Digital Twins: Leveraging the Physical and Virtual Worlds for Hybrid Sim2Real Transition of Multi-Agent Reinforcement Learning Policies
• MOB-Net: Limb-modularized Uncertainty Torque Learning of Humanoids for Sensorless External Torque Estimation
• Visual Semantic Navigation with Real Robots
• Efficient Reinforcement Learning for Jumping Monopods
• Dexterous Pre-grasp Manipulation for Human-like Functional Categorical Grasping: Deep Reinforcement Learning and Grasp Representations
• Advanced turning maneuver of a multi-legged robot using pitchfork bifurcation
• Deep reinforcement learning for the control of conjugate heat transfer with application to workpiece cooling
• Reinforcement Learning Control for Autonomous Hydraulic Material Handling Machines with Underactuated Tools
• Central pattern generators evolved for real-time adaptation to rhythmic stimuli
• Learning of Balance Controller Considering Changes in Body State for Musculoskeletal Humanoids
• Learning to Locomote with Deep Neural-Network and CPG-based Control in a Soft Snake Robot
• Gait Library Synthesis for Quadruped Robots via Augmented Random Search
• Learning to Manipulate Object Collections Using Grounded State Representations
• A Learnable Safety Measure
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• Machine-learning Based Extraction of the Short-Range Part of the Interaction in Non-contact Atomic Force Microscopy
• Proportional integral derivative controller assisted reinforcement learning for path following by autonomous underwater vehicles
• General Robot Dynamics Learning and Gen2Real
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds
• Efficient Learning of Control Policies for Robust Quadruped Bounding using Pretrained Neural Networks
• Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance
• Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning
• Learning Task Space Actions for Bipedal Locomotion
• Scalable sim-to-real transfer of soft robot designs
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
• Phoebe: Reuse-Aware Online Caching with Reinforcement Learning for Emerging Storage Models
• Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems
• DreamWaQ++: Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learning
• Flying Through a Narrow Gap Using End-to-end Deep Reinforcement Learning Augmented with Curriculum Learning and Sim2Real
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning
• Deep Visual MPC-Policy Learning for Navigation
• Lyapunov-stable neural-network control
• Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?
• Benchmarking Population-Based Reinforcement Learning across Robotic Tasks with GPU-Accelerated Simulation
• Reinforcement Learning with Lie Group Orientations for Robotics
• Robust Impedance Control for Dexterous Interaction Using Fractal Impedance Controller with IK-Optimisation
• Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach
• Vision-Proprioception Fusion with Mamba2 in End-to-End Reinforcement Learning for Motion Control
• Continuous Jumping of a Parallel Wire-Driven Monopedal Robot RAMIEL Using Reinforcement Learning
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Auditing Robot Learning for Safety and Compliance during Deployment
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Learning Agile Locomotion Skills with a Mentor
• Protective Policy Transfer
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• How does the structure embedded in learning policy affect learning quadruped locomotion?
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Active Model Learning using Informative Trajectories for Improved Closed-Loop Control on Real Robots
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Advances, challenges, and opportunities for legged robots
• Regret Bounds for Adaptive Nonlinear Control
• Improving the Robustness of Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Agile perceptive multi-skill locomotion for quadrupedal robots in the wild
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• A Kernel Two-sample Test for Dynamical Systems
• BinWalker: Development and Field Evaluation of a Quadruped Manipulator Platform for Sustainable Litter Collection
• Two-stage training algorithm for AI robot soccer
• Quantum spatial best-arm identification via quantum walks
• Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments
• LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba
• Out-of-the-box channel pruned networks
• SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics
• Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models
• Robustifying Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Adaptive Identification of Legged Robotic Kinematic Structure
• Quadruped Locomotion on Non-Rigid Terrain using Reinforcement Learning
• Energy-Efficient Omnidirectional Locomotion for Wheeled Quadrupeds via Predictive Energy-Aware Nominal Gait Selection
• On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration
• Contact Planning for the ANYmal Quadruped Robot using an Acyclic Reachability-Based Planner
• Task-Informed Fidelity Management for Speeding Up Robotics Simulation
• Improved Reinforcement Learning Coordinated Control of a Mobile Manipulator using Joint Clamping
• Generating Diverse Challenging Terrains for Legged Robots Using Quality-Diversity Algorithm
• Learn to Swim: Data-Driven LSTM Hydrodynamic Model for Quadruped Robot Gait Optimization