Visual Adversarial Imitation Learning using Variational Models

Abstract

Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including representation learning for visual observations, sample complexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model-based adversarial imitation learning (V-MAIL) algorithm. The model-based approach provides a strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling on-policy learning. Through experiments involving several vision-based locomotion and manipulation tasks, we find that V-MAIL learns successful visuomotor policies in a sample-efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can learn new tasks from visual demonstrations without any additional environment interactions. All results including videos can be found online at \url{https://sites.google.com/view/variational-mail}.

References in corpus (21)

• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Conservative Q-Learning for Offline Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• MOPO: Model-based Offline Policy Optimization
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Imitation Learning as $f$-Divergence Minimization
• Imitation Learning via Off-Policy Distribution Matching
• Model-based Adversarial Imitation Learning
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Feedback in Imitation Learning: The Three Regimes of Covariate Shift
• Offline Reinforcement Learning from Images with Latent Space Models
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification