Meta Learning Shared Hierarchies

Abstract

We develop a metalearning approach for learning hierarchically structured policies, improving sample efficiency on unseen tasks through the use of shared primitives---policies that are executed for large numbers of timesteps. Specifically, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies. We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks. We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies. We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes. We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

References in corpus (2)

• Categorical Reparameterization with Gumbel-Softmax
• Autonomous Extracting a Hierarchical Structure of Tasks in Reinforcement Learning and Multi-task Reinforcement Learning

Cited by in corpus (49)

• Character Controllers Using Motion VAEs
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Reinforcement Learning for Angle-Only Intercept Guidance of Maneuvering Targets
• Adaptive Guidance and Integrated Navigation with Reinforcement Meta-Learning
• Seeker based Adaptive Guidance via Reinforcement Meta-Learning Applied to Asteroid Close Proximity Operations
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Six Degree-of-Freedom Body-Fixed Hovering over Unmapped Asteroids via LIDAR Altimetry and Reinforcement Meta-Learning
• Small Sample Learning in Big Data Era
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Modular meta-learning
• Reinforcement Meta-Learning for Interception of Maneuvering Exoatmospheric Targets with Parasitic Attitude Loop
• Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Learning to Compose Skills
• Generalized Hindsight for Reinforcement Learning
• Hierarchical Reinforcement Learning for Multi-agent MOBA Game
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Context-Aware Policy Reuse
• VFunc: a Deep Generative Model for Functions
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Discovery of Options via Meta-Learned Subgoals
• Towards Autonomous Pipeline Inspection with Hierarchical Reinforcement Learning
• Hierarchical Learning for Modular Robots
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• Contextualizing Enhances Gradient Based Meta Learning
• ROAM: Recurrently Optimizing Tracking Model
• Optimal Options for Multi-Task Reinforcement Learning Under Time Constraints
• Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
• Population-Based Evolution Optimizes a Meta-Learning Objective
• A Deep Reinforcement Learning Architecture for Multi-stage Optimal Control
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Learning to Cope with Adversarial Attacks
• Temporal-adaptive Hierarchical Reinforcement Learning
• Reinforcement Learning Experience Reuse with Policy Residual Representation
• Extended Radial Basis Function Controller for Reinforcement Learning
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Adaptation-Agnostic Meta-Training
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Developing cooperative policies for multi-stage tasks
• Hyperbolic Embeddings for Learning Options in Hierarchical Reinforcement Learning
• Learning Shared Dynamics with Meta-World Models
• On mechanisms for transfer using landmark value functions in multi-task lifelong reinforcement learning