Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones

Abstract

Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration. We propose Recovery RL, an algorithm which navigates this tradeoff by (1) leveraging offline data to learn about constraint violating zones before policy learning and (2) separating the goals of improving task performance and constraint satisfaction across two policies: a task policy that only optimizes the task reward and a recovery policy that guides the agent to safety when constraint violation is likely. We evaluate Recovery RL on 6 simulation domains, including two contact-rich manipulation tasks and an image-based navigation task, and an image-based obstacle avoidance task on a physical robot. We compare Recovery RL to 5 prior safe RL methods which jointly optimize for task performance and safety via constrained optimization or reward shaping and find that Recovery RL outperforms the next best prior method across all domains. Results suggest that Recovery RL trades off constraint violations and task successes 2 - 20 times more efficiently in simulation domains and 3 times more efficiently in physical experiments. See https://tinyurl.com/rl-recovery for videos and supplementary material.

RA-L and ICRA 2021. First two authors contributed equally

References in corpus (5)

• Lyapunov-based Safe Policy Optimization for Continuous Control
• Deep Dynamics Models for Learning Dexterous Manipulation
• Worst Cases Policy Gradients
• Learning to be Safe: Deep RL with a Safety Critic
• Goal-Aware Prediction: Learning to Model What Matters

Cited by in corpus (7)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Reinforcement Learning by Guided Safe Exploration
• Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling