Robust Imitation of Diverse Behaviors

Abstract

Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

References in corpus (17)

• Continuous control with deep reinforcement learning
• Conditional Generative Adversarial Nets
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NIPS 2016 Tutorial: Generative Adversarial Networks
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• A note on the evaluation of generative models
• Generative Adversarial Imitation Learning
• One-Shot Imitation Learning
• Least Squares Generative Adversarial Networks
• Learning human behaviors from motion capture by adversarial imitation
• Third-Person Imitation Learning
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Boundary-Seeking Generative Adversarial Networks
• MAGAN: Margin Adaptation for Generative Adversarial Networks
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Model-based Adversarial Imitation Learning

Cited by in corpus (6)

• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Deep reinforcement learning for guidewire navigation in coronary artery phantom
• DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations
• Fast Adaptation via Policy-Dynamics Value Functions
• Data-Driven Stochastic Motion Evaluation and Optimization with Image by Spatially-Aligned Temporal Encoding
• PODNet: A Neural Network for Discovery of Plannable Options