Task-Relevant Adversarial Imitation Learning

Abstract

We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performance. We analyze this problem in detail and propose a solution that outperforms standard Generative Adversarial Imitation Learning (GAIL). Our proposed method, Task-Relevant Adversarial Imitation Learning (TRAIL), uses constrained discriminator optimization to learn informative rewards. In comprehensive experiments, we show that TRAIL can solve challenging robotic manipulation tasks from pixels by imitating human operators without access to any task rewards, and clearly outperforms comparable baseline imitation agents, including those trained via behaviour cloning and conventional GAIL.

arXiv commentAccepted to CoRL 2020 (see presentation here: https://youtu.be/ZgQvFGuEgFU )

References in corpus (4)

• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Distributed Distributional Deterministic Policy Gradients

Cited by in corpus (9)

• Imitating Interactive Intelligence
• Positive-Unlabeled Reward Learning
• Semi-supervised reward learning for offline reinforcement learning
• Shaking the foundations: delusions in sequence models for interaction and control
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Domain-Robust Visual Imitation Learning with Mutual Information Constraints
• Adversarially Regularized Policy Learning Guided by Trajectory Optimization
• Towards intervention-centric causal reasoning in learning agents