CompILE: Compositional Imitation Learning and Execution

Abstract

We introduce Compositional Imitation Learning and Execution (CompILE): a framework for learning reusable, variable-length segments of hierarchically-structured behavior from demonstration data. CompILE uses a novel unsupervised, fully-differentiable sequence segmentation module to learn latent encodings of sequential data that can be re-composed and executed to perform new tasks. Once trained, our model generalizes to sequences of longer length and from environment instances not seen during training. We evaluate CompILE in a challenging 2D multi-task environment and a continuous control task, and show that it can find correct task boundaries and event encodings in an unsupervised manner. Latent codes and associated behavior policies discovered by CompILE can be used by a hierarchical agent, where the high-level policy selects actions in the latent code space, and the low-level, task-specific policies are simply the learned decoders. We found that our CompILE-based agent could learn given only sparse rewards, where agents without task-specific policies struggle.

arXiv commentICML (2019)

References in corpus (16)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• DeepMind Control Suite
• Composing graphical models with neural networks for structured representations and fast inference
• Modular Multitask Reinforcement Learning with Policy Sketches
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• DART: Noise Injection for Robust Imitation Learning
• Multi-Level Discovery of Deep Options
• Learning Abstract Options
• Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
• Latent Sequence Decompositions
• Sequence Modeling via Segmentations
• TACO: Learning Task Decomposition via Temporal Alignment for Control

Cited by in corpus (3)

• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Product Kanerva Machines: Factorized Bayesian Memory
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation