IPO: Interior-point Policy Optimization under Constraints

Abstract

In this paper, we study reinforcement learning (RL) algorithms to solve real-world decision problems with the objective of maximizing the long-term reward as well as satisfying cumulative constraints. We propose a novel first-order policy optimization method, Interior-point Policy Optimization (IPO), which augments the objective with logarithmic barrier functions, inspired by the interior-point method. Our proposed method is easy to implement with performance guarantees and can handle general types of cumulative multiconstraint settings. We conduct extensive evaluations to compare our approach with state-of-the-art baselines. Our algorithm outperforms the baseline algorithms, in terms of reward maximization and constraint satisfaction.

References in corpus (7)

• Trust Region Policy Optimization
• Safe Exploration in Continuous Action Spaces
• Learning Dexterous In-Hand Manipulation
• Reward Constrained Policy Optimization
• Challenges of Real-World Reinforcement Learning
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

Cited by in corpus (4)

• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning