Generative Adversarial Imitation from Observation

Abstract

Imitation from observation (IfO) is the problem of learning directly from state-only demonstrations without having access to the demonstrator's actions. The lack of action information both distinguishes IfO from most of the literature in imitation learning, and also sets it apart as a method that may enable agents to learn from a large set of previously inapplicable resources such as internet videos. In this paper, we propose both a general framework for IfO approaches and also a new IfO approach based on generative adversarial networks called generative adversarial imitation from observation (GAIfO). We conduct experiments in two different settings: (1) when demonstrations consist of low-dimensional, manually-defined state features, and (2) when demonstrations consist of high-dimensional, raw visual data. We demonstrate that our approach performs comparably to classical imitation learning approaches (which have access to the demonstrator's actions) and significantly outperforms existing imitation from observation methods in high-dimensional simulation environments.

References in corpus (9)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• One-Shot Visual Imitation Learning via Meta-Learning
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning human behaviors from motion capture by adversarial imitation
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning

Cited by in corpus (42)

• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Imitating Latent Policies from Observation
• Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
• Semantic Visual Navigation by Watching YouTube Videos
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• State Alignment-based Imitation Learning
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Imitation Learning: Progress, Taxonomies and Challenges
• State-only Imitation with Transition Dynamics Mismatch
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• Hindsight Generative Adversarial Imitation Learning
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Counterfactual experience augmented off-policy reinforcement learning
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Sample-efficient Adversarial Imitation Learning from Observation
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Reinforced Imitation in Heterogeneous Action Space
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Discovering Fatigued Movements for Virtual Character Animation
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• MobILE: Model-Based Imitation Learning From Observation Alone
• Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
• Data Driven Aircraft Trajectory Prediction with Deep Imitation Learning
• Differentiable Implicit Layers
• Programmatic Imitation Learning from Unlabeled and Noisy Demonstrations
• Perceptual Values from Observation
• Provably Efficient Third-Person Imitation from Offline Observation
• Learning from Imperfect Demonstrations from Agents with Varying Dynamics
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Reparameterized Variational Divergence Minimization for Stable Imitation
• RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning
• Relational Mimic for Visual Adversarial Imitation Learning
• Injective State-Image Mapping facilitates Visual Adversarial Imitation Learning
• Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• Towards Learning to Imitate from a Single Video Demonstration
• HILONet: Hierarchical Imitation Learning from Non-Aligned Observations
• On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration
• Towards intervention-centric causal reasoning in learning agents
• Learning What To Do by Simulating the Past