Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

Abstract

The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.

Accepted as conference paper at ICLR'19

Cited by in corpus (11)

• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
• CompILE: Compositional Imitation Learning and Execution
• Adversarial Option-Aware Hierarchical Imitation Learning
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
• Inverse Contextual Bandits: Learning How Behavior Evolves over Time
• Learning the Solution Manifold in Optimization and Its Application in Motion Planning
• oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
• GRP Model for Sensorimotor Learning