Learning and Transfer of Modulated Locomotor Controllers

Abstract

We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at https://youtu.be/sboPYvhpraQ

arXiv commentSupplemental video available at https://youtu.be/sboPYvhpraQ

References in corpus (5)

• Continuous control with deep reinforcement learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• End-to-End Training of Deep Visuomotor Policies
• Learning Continuous Control Policies by Stochastic Value Gradients

Cited by in corpus (56)

• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Emergence of Locomotion Behaviours in Rich Environments
• Reinforcement Learning with Deep Energy-Based Policies
• Multi-Task Learning with Deep Neural Networks: A Survey
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Hybrid Reward Architecture for Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• RUDDER: Return Decomposition for Delayed Rewards
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• A survey on intrinsic motivation in reinforcement learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Dynamics-Aware Unsupervised Discovery of Skills
• Hierarchical visuomotor control of humanoids
• Latent Space Policies for Hierarchical Reinforcement Learning
• Multi-Level Discovery of Deep Options
• Relative Entropy Regularized Policy Iteration
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Universal Successor Features Approximators
• Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Behavior Priors for Efficient Reinforcement Learning
• UniCon: Universal Neural Controller For Physics-based Character Motion
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• Data-efficient Hindsight Off-policy Option Learning
• Deep Reinforcement Learning to Acquire Navigation Skills for Wheel-Legged Robots in Complex Environments
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• Separation of Concerns in Reinforcement Learning
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Discovery of Options via Meta-Learned Subgoals
• Hierarchical principles of embodied reinforcement learning: A review
• Following Instructions by Imagining and Reaching Visual Goals
• Automata-Guided Hierarchical Reinforcement Learning for Skill Composition
• Towards Autonomous Pipeline Inspection with Hierarchical Reinforcement Learning
• Diversity-Driven Extensible Hierarchical Reinforcement Learning
• Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Multi-task Learning with Gradient Guided Policy Specialization
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Object-oriented state editing for HRL
• Options Discovery with Budgeted Reinforcement Learning
• Temporal Abstraction in Reinforcement Learning with the Successor Representation
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Expanding Motor Skills through Relay Neural Networks
• Training Transition Policies via Distribution Matching for Complex Tasks
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Neural Embedding for Physical Manipulations