Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

Abstract

Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updates parameters in primal and dual spaces in turn. Existing methods for CMDPs only use on-policy data for dual updates, which results in sample inefficiency and slow convergence. In this paper, we propose a policy search method for CMDPs called Accelerated Primal-Dual Optimization (APDO), which incorporates an off-policy trained dual variable in the dual update procedure while updating the policy in primal space with on-policy likelihood ratio gradient. Experimental results on a simulated robot locomotion task show that APDO achieves better sample efficiency and faster convergence than state-of-the-art approaches for CMDPs.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Constrained Policy Optimization
• Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
• Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Cited by in corpus (11)

• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• Secure Deep Reinforcement Learning for Dynamic Resource Allocation in Wireless MEC Networks
• IPO: Interior-point Policy Optimization under Constraints
• Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs
• A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants
• Deep Reinforcement Learning for Tropical Air Free-Cooled Data Center Control
• Policy Gradients for Probabilistic Constrained Reinforcement Learning
• Learning-based Prediction, Rendering and Transmission for Interactive Virtual Reality in RIS-Assisted Terahertz Networks
• Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes
• Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning