Projection-Based Constrained Policy Optimization

Abstract

We consider the problem of learning control policies that optimize a reward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm, Projection-Based Constrained Policy Optimization (PCPO). This is an iterative method for optimizing policies in a two-step process: the first step performs a local reward improvement update, while the second step reconciles any constraint violation by projecting the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on reward improvement, and an upper bound on constraint violation, for each policy update. We further characterize the convergence of PCPO based on two different metrics: $\normltwo$ norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that PCPO achieves superior performance, averaging more than 3.5 times less constraint violation and around 15\% higher reward compared to state-of-the-art methods.

International Conference on Learning Representations (ICLR) 2020

References in corpus (1)

• Lyapunov-based Safe Policy Optimization for Continuous Control

Cited by in corpus (13)

• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk
• Enforcing robust control guarantees within neural network policies
• Safe Reinforcement Learning with Natural Language Constraints
• Differentiable Trust Region Layers for Deep Reinforcement Learning
• Reinforcement Learning by Guided Safe Exploration
• Policy Gradients for Probabilistic Constrained Reinforcement Learning
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
• Safe Distributional Reinforcement Learning