Constrained Policy Optimization

Abstract

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

Accepted to ICML 2017

References in corpus (4)

• Continuous control with deep reinforcement learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Deep Reinforcement Learning for Autonomous Driving

Cited by in corpus (60)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Safe Exploration in Continuous Action Spaces
• Reward Constrained Policy Optimization
• Towards Long-term Fairness in Recommendation
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Scalable agent alignment via reward modeling: a research direction
• AI Safety Gridworlds
• Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Conservative Safety Critics for Exploration
• Deep Reinforcement Learning for Six Degree-of-Freedom Planetary Powered Descent and Landing
• A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Constrained Upper Confidence Reinforcement Learning
• Stability-certified reinforcement learning: A control-theoretic perspective
• Mutual Alignment Transfer Learning
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• A Primal-Dual Approach to Constrained Markov Decision Processes
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• Nearly Minimax Optimal Reward-free Reinforcement Learning
• Improving the Generalization of End-to-End Driving through Procedural Generation
• Lyapunov Barrier Policy Optimization
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs
• Scalable Synthesis of Verified Controllers in Deep Reinforcement Learning
• Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization
• Reinforcement Learning Based Safe Decision Making for Highway Autonomous Driving
• Safe Value Functions
• Constrained Model-Free Reinforcement Learning for Process Optimization
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Formal Language Constraints for Markov Decision Processes
• Practical Risk Measures in Reinforcement Learning
• Reconnaissance and Planning algorithm for constrained MDP
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric
• Constrained Exploration and Recovery from Experience Shaping
• Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples
• Probably Approximately Correct Constrained Learning
• Learning Adaptive Display Exposure for Real-Time Advertising
• Sufficiently Accurate Model Learning
• Tomography Based Learning for Load Distribution through Opaque Networks
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• Assisted Robust Reward Design
• Constraint Estimation and Derivative-Free Recovery for Robot Learning from Demonstrations
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• Safety-guaranteed Reinforcement Learning based on Multi-class Support Vector Machine
• Differentiable Robust LQR Layers