Lyapunov-based Safe Policy Optimization for Continuous Control

Abstract

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction. Videos of the experiments can be found in the following link: https://drive.google.com/file/d/1pzuzFqWIE710bE2U6DmS59AfRzqK2Kek/view?usp=sharing.

References in corpus (10)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Safe Exploration in Continuous Action Spaces
• OptNet: Differentiable Optimization as a Layer in Neural Networks
• Constrained Policy Optimization
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow

Cited by in corpus (50)

• Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
• Learning to Walk in the Real World with Minimal Human Effort
• Projection-Based Constrained Policy Optimization
• First Order Constrained Optimization in Policy Space
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Safe Reinforcement Learning via Curriculum Induction
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Conservative Safety Critics for Exploration
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Road Traffic Law Adaptive Decision-making for Self-Driving Vehicles
• Learning to be Safe: Deep RL with a Safety Critic
• Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
• IPO: Interior-point Policy Optimization under Constraints
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Multi-Agent Constrained Policy Optimisation
• ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllers
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning
• Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces
• Safe Interactive Model-Based Learning
• Actor-Critic Reinforcement Learning for Control with Stability Guarantee
• Lyapunov Barrier Policy Optimization
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Safe Reinforcement Learning with Natural Language Constraints
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Differentiable Trust Region Layers for Deep Reinforcement Learning
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Provably Correct Training of Neural Network Controllers Using Reachability Analysis
• A Risk-Sensitive Approach to Policy Optimization
• Data-driven Outer-Loop Control Using Deep Reinforcement Learning for Trajectory Tracking
• Reconnaissance and Planning algorithm for constrained MDP
• Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee
• RMP2: A Structured Composable Policy Class for Robot Learning
• Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• A Dynamic Penalty Function Approach for Constraints-Handling in Reinforcement Learning
• Rarely-switching linear bandits: optimization of causal effects for the real world
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
• Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics
• Reparameterized Variational Divergence Minimization for Stable Imitation
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Model-Based Actor-Critic with Chance Constraint for Stochastic System
• Lyapunov-based uncertainty-aware safe reinforcement learning