Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning

Abstract

We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by representation and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to an optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the planning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. Video: https://youtu.be/HQsQysUWOhg

arXiv commentICRA 2021, the first two authors contributed equally

References in corpus (15)

• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Data-Efficient Hierarchical Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Adaptive importance sampling for control and inference
• Neural probabilistic motor primitives for humanoid control
• Dynamics-Aware Unsupervised Discovery of Skills
• Path Integral Networks: End-to-End Differentiable Optimal Control
• Learning Latent Plans from Play
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems