Safe Exploration by Solving Early Terminated MDP

Abstract

Safe exploration is crucial for the real-world application of reinforcement learning (RL). Previous works consider the safe exploration problem as Constrained Markov Decision Process (CMDP), where the policies are being optimized under constraints. However, when encountering any potential dangers, human tends to stop immediately and rarely learns to behave safely in danger. Motivated by human learning, we introduce a new approach to address safe RL problems under the framework of Early Terminated MDP (ET-MDP). We first define the ET-MDP as an unconstrained MDP with the same optimal value function as its corresponding CMDP. An off-policy algorithm based on context models is then proposed to solve the ET-MDP, which thereby solves the corresponding CMDP with better asymptotic performance and improved learning efficiency. Experiments on various CMDP tasks show a substantial improvement over previous methods that directly solve CMDP.

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• On First-Order Meta-Learning Algorithms
• Safe Exploration in Continuous Action Spaces
• Benchmarking Model-Based Reinforcement Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Open-Sourced Reinforcement Learning Environments for Surgical Robotics
• Safe Reinforcement Learning in Constrained Markov Decision Processes
• Conservative Safety Critics for Exploration
• Learning to be Safe: Deep RL with a Safety Critic
• Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
• Lyapunov Barrier Policy Optimization
• Constrained Exploration and Recovery from Experience Shaping