Goal-conditioned Imitation Learning

Abstract

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available at https://sites.google.com/view/goalconditioned-il/.

Published at NeurIPS 2019

References in corpus (14)

• Continuous control with deep reinforcement learning
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Visual Reinforcement Learning with Imagined Goals
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Third-Person Imitation Learning
• Reinforcement Learning from Imperfect Demonstrations
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Self-supervised Learning of Image Embedding for Continuous Control
• Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
• Generative predecessor models for sample-efficient imitation learning

Cited by in corpus (14)

• Learning to Reach Goals via Iterated Supervised Learning
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles
• Transformers for One-Shot Visual Imitation
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control
• Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning
• Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning
• Addressing reward bias in Adversarial Imitation Learning with neutral reward functions
• LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning
• Learning to Play by Imitating Humans
• No Need for Interactions: Robust Model-Based Imitation Learning using Neural ODE