Policies Modulating Trajectory Generators

Abstract

We propose an architecture for learning complex controllable behaviors by having simple Policies Modulate Trajectory Generators (PMTG), a powerful combination that can provide both memory and prior knowledge to the controller. The result is a flexible architecture that is applicable to a class of problems with periodic motion for which one has an insight into the class of trajectories that might lead to a desired behavior. We illustrate the basics of our architecture using a synthetic control problem, then go on to learn speed-controlled locomotion for a quadrupedal robot by using Deep Reinforcement Learning and Evolutionary Strategies. We demonstrate that a simple linear policy, when paired with a parametric Trajectory Generator for quadrupedal gaits, can induce walking behaviors with controllable speed from 4-dimensional IMU observations alone, and can be learned in under 1000 rollouts. We also transfer these policies to a real robot and show locomotion with controllable forward velocity.

Cited by in corpus (22)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning
• Imitation Learning from MPC for Quadrupedal Multi-Gait Control
• Data Efficient Reinforcement Learning for Legged Robots
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• RMA: Rapid Motor Adaptation for Legged Robots
• Provably Robust Blackbox Optimization for Reinforcement Learning
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• Fast and Efficient Locomotion via Learned Gait Transitions
• Zero-Shot Terrain Generalization for Visual Locomotion Policies
• Robust Quadrupedal Locomotion on Sloped Terrains: A Linear Policy Approach
• Learning Agile Locomotion via Adversarial Training
• Dynamics and Domain Randomized Gait Modulation with Bezier Curves for Sim-to-Real Legged Locomotion
• Gait Library Synthesis for Quadruped Robots via Augmented Random Search
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Reinforcement Learning with Chromatic Networks for Compact Architecture Search
• Discovering Diverse Athletic Jumping Strategies
• Learning Agile Locomotion Skills with a Mentor
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization