Safe Exploration in Continuous Action Spaces

Abstract

We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be violated. We show how to exploit the typically smooth dynamics of these systems and enable RL algorithms to never violate constraints during learning. Our technique is to directly add to the policy a safety layer that analytically solves an action correction formulation per each state. The novelty of obtaining an elegant closed-form solution is attained due to a linearized model, learned on past trajectories consisting of arbitrary actions. This is to mimic the real-world circumstances where data logs were generated with a behavior policy that is implausible to describe mathematically; such cases render the known safety-aware off-policy methods inapplicable. We demonstrate the efficacy of our approach on new representative physics-based environments, and prevail where reward shaping fails by maintaining zero constraint violations.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• OptNet: Differentiable Optimization as a Layer in Neural Networks
• Constrained Policy Optimization
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World

Cited by in corpus (80)

• Exploration in Deep Reinforcement Learning: A Survey
• Safe Reinforcement Learning Using Robust MPC
• Challenges of Real-World Reinforcement Learning
• Reward Constrained Policy Optimization
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Learning to Walk in the Real World with Minimal Human Effort
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Conservative Safety Critics for Exploration
• Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions
• Safe Reinforcement Learning Using Black-Box Reachability Analysis
• Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
• IPO: Interior-point Policy Optimization under Constraints
• Penalizing side effects using stepwise relative reachability
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Adaptive Power System Emergency Control using Deep Reinforcement Learning
• On the model-based stochastic value gradient for continuous reinforcement learning
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• Accelerated Deep Reinforcement Learning Based Load Shedding for Emergency Voltage Control
• Neurosymbolic Reinforcement Learning with Formally Verified Exploration
• Multi-Objective Predictive Taxi Dispatch via Network Flow Optimization
• Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraints
• A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants
• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces
• Constrained Reinforcement Learning Has Zero Duality Gap
• Conformal Predictive Safety Filter for RL Controllers in Dynamic Environments
• Learning Human Objectives by Evaluating Hypothetical Behavior
• Lyapunov Barrier Policy Optimization
• Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Safety Filtering for Reinforcement Learning-based Adaptive Cruise Control
• Imitation-Projected Programmatic Reinforcement Learning
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertainty
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Learning Autonomous Race Driving with Action Mapping Reinforcement Learning
• Reward Constrained Interactive Recommendation with Natural Language Feedback
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Data-driven Outer-Loop Control Using Deep Reinforcement Learning for Trajectory Tracking
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Safe Driving via Expert Guided Policy Optimization
• Formal Language Constraints for Markov Decision Processes
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Safety design concepts for statistical machine learning components toward accordance with functional safety standards
• Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
• RMP2: A Structured Composable Policy Class for Robot Learning
• Safe Policy Improvement in Constrained Markov Decision Processes
• Stable Reinforcement Learning with Unbounded State Space
• Learning to Generate All Feasible Actions
• Safe Exploration by Solving Early Terminated MDP
• Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
• Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
• Safe Distributional Reinforcement Learning
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• CLARA: A Constrained Reinforcement Learning Based Resource Allocation Framework for Network Slicing
• Tomography Based Learning for Load Distribution through Opaque Networks
• Model-Based Actor-Critic with Chance Constraint for Stochastic System
• Deep Reinforcement Learning with Adjustments
• Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy RL
• Price of Safety in Linear Best Arm Identification
• Let's Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety
• Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning
• SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation
• FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimize
• Deep Reinforcement Learning with Symmetric Prior for Predictive Power Allocation to Mobile Users
• Accelerating Deep Reinforcement Learning With the Aid of Partial Model: Energy-Efficient Predictive Video Streaming
• Automatic Exploration Process Adjustment for Safe Reinforcement Learning with Joint Chance Constraint Satisfaction