Imitating Latent Policies from Observation

Abstract

In this paper, we describe a novel approach to imitation learning that infers latent policies directly from state observations. We introduce a method that characterizes the causal effects of latent actions on observations while simultaneously predicting their likelihood. We then outline an action alignment procedure that leverages a small amount of environment interactions to determine a mapping between the latent and real-world actions. We show that this corrected labeling can be used for imitating the observed behavior, even though no expert actions are given. We evaluate our approach within classic control environments and a platform game and demonstrate that it performs better than standard approaches. Code for this work is available at https://github.com/ashedwards/ILPO.

Accepted to ICML 2019

References in corpus (10)

• Toward Multimodal Image-to-Image Translation
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Quantifying Generalization in Reinforcement Learning
• Third-Person Imitation Learning
• Recurrent Environment Simulators
• Generative Adversarial Imitation from Observation
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Forward-Backward Reinforcement Learning

Cited by in corpus (22)

• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Primal Wasserstein Imitation Learning
• Semantic Visual Navigation by Watching YouTube Videos
• Provably Efficient Imitation Learning from Observation Alone
• State Alignment-based Imitation Learning
• On a probabilistic approach to synthesize control policies from example datasets
• Learning Navigation Subroutines from Egocentric Videos
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Imitation Learning: Progress, Taxonomies and Challenges
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Provable Representation Learning for Imitation Learning via Bi-level Optimization
• MobILE: Model-Based Imitation Learning From Observation Alone
• Perceptual Values from Observation
• Linear Disentangled Representations and Unsupervised Action Estimation
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• Imitating Unknown Policies via Exploration
• HILONet: Hierarchical Imitation Learning from Non-Aligned Observations
• Towards intervention-centric causal reasoning in learning agents