Safe Reinforcement Learning via Curriculum Induction

Abstract

In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. In such settings, the agent needs to behave safely not only after but also while learning. To achieve this, existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations during exploration with high probability, but both the probabilistic guarantees and the smoothness assumptions inherent in the priors are not viable in many scenarios of interest such as autonomous driving. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor that saves the agent from violating constraints during learning. In this model, we introduce the monitor that neither needs to know how to do well at the task the agent is learning nor needs to know how the environment works. Instead, it has a library of reset controllers that it activates when the agent starts behaving dangerously, preventing it from doing damage. Crucially, the choices of which reset controller to apply in which situation affect the speed of agent learning. Based on observing agents' progress, the teacher itself learns a policy for choosing the reset controllers, a curriculum, to optimize the agent's final policy reward. Our experiments use this framework in two environments to induce curricula for safe and efficient learning.

References in corpus (14)

• Safe Model-based Reinforcement Learning with Stability Guarantees
• Safe Controller Optimization for Quadrotors with Gaussian Processes
• Challenges of Real-World Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Meta-Learning: A Survey
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Multi-Armed Bandits for Intelligent Tutoring Systems
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Approximate Policy Iteration Schemes: A Comparison
• Transfer from Multiple MDPs
• Safe Exploration for Interactive Machine Learning

Cited by in corpus (14)

• Autonomous Drone Racing: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles
• A Survey of Reinforcement Learning for Optimization in Automation
• Safe Reinforcement Learning with Natural Language Constraints
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Reinforcement Learning by Guided Safe Exploration
• Meta Automatic Curriculum Learning
• Toddler-Guidance Learning: Impacts of Critical Period on Multimodal AI Agents
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Safe Reinforcement Learning with Linear Function Approximation
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimize
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning