Learning to Reach Goals via Iterated Supervised Learning

Abstract

Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor. In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function. In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the final state in that same trajectory. We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch. Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy. We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks.

First two authors contributed equally. Code available at https://github.com/dibyaghosh/gcsl

References in corpus (18)

• Continuous control with deep reinforcement learning
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Visual Reinforcement Learning with Imagined Goals
• Deep Q-learning from Demonstrations
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Self-Imitation Learning
• Deep Reinforcement Learning and the Deadly Triad
• Semi-parametric Topological Memory for Navigation
• Hindsight policy gradients
• Goal-conditioned Imitation Learning
• Learning Latent Plans from Play

Cited by in corpus (12)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Training Agents using Upside-Down Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Reward-Conditioned Policies
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Planning from Pixels using Inverse Dynamics Models
• Zeroth-Order Supervised Policy Improvement
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation
• Evolutionary Stochastic Policy Distillation