Reward Constrained Policy Optimization

Abstract

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

arXiv commentAccepted as a poster to ICLR 2019

References in corpus (7)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Safe Exploration in Continuous Action Spaces
• Simple random search provides a competitive approach to reinforcement learning
• AI Safety Gridworlds
• Constrained Policy Optimization
• Variance Adjusted Actor Critic Algorithms

Cited by in corpus (76)

• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Challenges of Real-World Reinforcement Learning
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Projection-Based Constrained Policy Optimization
• First Order Constrained Optimization in Policy Space
• An empirical investigation of the challenges of real-world reinforcement learning
• Reinforcement Learning with Convex Constraints
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• On Reward-Free Reinforcement Learning with Linear Function Approximation
• Conservative Safety Critics for Exploration
• Safe Reinforcement Learning Using Black-Box Reachability Analysis
• Constrained Deep Reinforcement Based Functional Split Optimization in Virtualized RANs
• Road Traffic Law Adaptive Decision-making for Self-Driving Vehicles
• Reward-Free Exploration for Reinforcement Learning
• TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning
• IPO: Interior-point Policy Optimization under Constraints
• A Bayesian Approach to Robust Reinforcement Learning
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• Lexicographic Multi-Objective Reinforcement Learning
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk
• Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs
• A Primal-Dual Approach to Constrained Markov Decision Processes
• Multi-Objective Predictive Taxi Dispatch via Network Flow Optimization
• Deep Constrained Q-learning
• Constrained Combinatorial Optimization with Reinforcement Learning
• GIN: Graph-based Interaction-aware Constraint Policy Optimization for Autonomous Driving
• Constrained episodic reinforcement learning in concave-convex and knapsack settings
• A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants
• ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning
• Nearly Minimax Optimal Reward-free Reinforcement Learning
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Constrained Reinforcement Learning Has Zero Duality Gap
• Reinforcement Learning for Multi-Objective and Constrained Markov Decision Processes
• Deep Reinforcement Learning for Tropical Air Free-Cooled Data Center Control
• Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs
• Safe Reinforcement Learning with Natural Language Constraints
• Inverse Constrained Reinforcement Learning
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Policy Gradients for Probabilistic Constrained Reinforcement Learning
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Reinforcement Learning by Guided Safe Exploration
• Reward Constrained Interactive Recommendation with Natural Language Feedback
• Learning Autonomous Race Driving with Action Mapping Reinforcement Learning
• Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• A Risk-Sensitive Approach to Policy Optimization
• Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Risk-Aware Continuous Control with Neural Contextual Bandits
• Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process
• Safe Reinforcement Learning with Linear Function Approximation
• Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost
• Learning Adaptive Display Exposure for Real-Time Advertising
• Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• Reward is enough for convex MDPs
• Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes
• Safe Distributional Reinforcement Learning
• CLARA: A Constrained Reinforcement Learning Based Resource Allocation Framework for Network Slicing
• Vision-driven UAV River Following: Benchmarking with Safe Reinforcement Learning
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
• Concave Utility Reinforcement Learning with Zero-Constraint Violations
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Let's Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety
• Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning
• LiveMap: Real-Time Dynamic Map in Automotive Edge Computing
• Scheduling and Power Control for Wireless Multicast Systems via Deep Reinforcement Learning