Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning

Abstract

Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.

arXiv commentVideos of our experiments are available at: https://sites.google.com/site/mlleavenotrace/

References in corpus (10)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Deep Exploration via Bootstrapped DQN
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Automatic Goal Generation for Reinforcement Learning Agents
• Reverse Curriculum Generation for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Learning to Fly by Crashing

Cited by in corpus (11)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Safety Critics for Exploration
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Avoiding Side Effects By Considering Future Tasks
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• Conservative Agency via Attainable Utility Preservation
• Reset-Free Lifelong Learning with Skill-Space Planning
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Protective Policy Transfer
• Learning the Arrow of Time