Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Abstract

Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment. To overcome this difficulty, we propose a "safety-critical adaptation" task setting: an agent first trains in non-safety-critical "source" environments such as in a simulator, before it adapts to the target environment where failures carry heavy costs. We propose a solution approach, CARL, that builds on the intuition that prior experience in diverse environments equips an agent to estimate risk, which in turn enables relative safety through risk-averse, cautious adaptation. CARL first employs model-based RL to train a probabilistic model to capture uncertainty about transition dynamics and catastrophic states across varied source environments. Then, when exploring a new safety-critical environment with unknown dynamics, the CARL agent plans to avoid actions that could lead to catastrophic states. In experiments on car driving, cartpole balancing, half-cheetah locomotion, and robotic object manipulation, CARL successfully acquires cautious exploration behaviors, yielding higher rewards with fewer failures than strong RL adaptation baselines. Website at https://sites.google.com/berkeley.edu/carl.

arXiv comment15 pages, 8 figures, ICML 2020. Website with code: https://sites.google.com/berkeley.edu/carl

Cited by in corpus (11)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Reset-Free Lifelong Learning with Skill-Space Planning
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Lyapunov Barrier Policy Optimization
• Reinforcement Learning by Guided Safe Exploration
• Safe Exploration by Solving Early Terminated MDP
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation
• Safety Aware Reinforcement Learning (SARL)
• Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy RL
• Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning