Injective State-Image Mapping facilitates Visual Adversarial Imitation Learning

Abstract

The growing use of virtual autonomous agents in applications like games and entertainment demands better control policies for natural-looking movements and actions. Unlike the conventional approach of hard-coding motion routines, we propose a deep learning method for obtaining control policies by directly mimicking raw video demonstrations. Previous methods in this domain rely on extracting low-dimensional features from expert videos followed by a separate hand-crafted reward estimation step. We propose an imitation learning framework that reduces the dependence on hand-engineered reward functions by jointly learning the feature extraction and reward estimation steps using Generative Adversarial Networks (GANs). Our main contribution in this paper is to show that under injective mapping between low-level joint state (angles and velocities) trajectories and corresponding raw video stream, performing adversarial imitation learning on video demonstrations is equivalent to learning from the state trajectories. Experimental results show that the proposed adversarial learning method from raw videos produces a similar performance to state-of-the-art imitation learning techniques while frequently outperforming existing hand-crafted video imitation methods. Furthermore, we show that our method can learn action policies by imitating video demonstrations on YouTube with similar performance to learned agents from true reward signals. Please see the supplementary video submission at https://ibm.biz/BdzzNA.

arXiv commentUpdated the paper to match with version accepted at IEEE MMSP 2019

References in corpus (9)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• One-Shot Visual Imitation Learning via Meta-Learning
• Learning human behaviors from motion capture by adversarial imitation
• Generative Adversarial Imitation from Observation
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Internal Model from Observations for Reward Shaping