Learning and Transfer of Modulated Locomotor Controllers

Abstract

We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at https://youtu.be/sboPYvhpraQ

Supplemental video available at https://youtu.be/sboPYvhpraQ

Cited by in corpus (23)

• Emergence of Locomotion Behaviours in Rich Environments
• Multi-Task Learning with Deep Neural Networks: A Survey
• Hybrid Reward Architecture for Reinforcement Learning
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Universal Successor Features Approximators
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• Behavior Priors for Efficient Reinforcement Learning
• UniCon: Universal Neural Controller For Physics-based Character Motion
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• Separation of Concerns in Reinforcement Learning
• Discovery of Options via Meta-Learned Subgoals
• Following Instructions by Imagining and Reaching Visual Goals
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Object-oriented state editing for HRL
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Options Discovery with Budgeted Reinforcement Learning
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Neural Embedding for Physical Manipulations