SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards

Abstract

Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r=+1 for matching the demonstrated action in a demonstrated state, and a constant reward of r=0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive results compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Recurrent World Models Facilitate Policy Evolution
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Deep Reinforcement Learning that Matters
• Maximum Entropy Deep Inverse Reinforcement Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Reinforcement Learning from Imperfect Demonstrations
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Cited by in corpus (25)

• Modeling Human Driving Behavior through Generative Adversarial Imitation Learning
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• Energy-Based Imitation Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Augmenting GAIL with BC for sample efficient imitation learning
• Soft Expert Reward Learning for Vision-and-Language Navigation
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Safe Driving via Expert Guided Policy Optimization
• Benders Cut Classification via Support Vector Machines for Solving Two-stage Stochastic Programs
• Reinforced Imitation Learning by Free Energy Principle
• Transferable Active Grasping and Real Embodied Dataset
• Discriminator Soft Actor Critic without Extrinsic Rewards
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• IQ-Learn: Inverse soft-Q Learning for Imitation
• Offline Inverse Reinforcement Learning
• Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
• Imitation with Neural Density Models