First Order Constrained Optimization in Policy Space

Abstract

In reinforcement learning, an agent attempts to learn high-performing behaviors through interacting with the environment, such behaviors are often quantified in the form of a reward function. However some aspects of behavior-such as ones which are deemed unsafe and to be avoided-are best captured through constraints. We propose a novel approach called First Order Constrained Optimization in Policy Space (FOCOPS) which maximizes an agent's overall reward while ensuring the agent satisfies a set of cost constraints. Using data generated from the current policy, FOCOPS first finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space. FOCOPS then projects the update policy back into the parametric policy space. Our approach has an approximate upper bound for worst-case constraint violation throughout training and is first-order in nature therefore simple to implement. We provide empirical evidence that our simple approach achieves better performance on a set of constrained robotics locomotive tasks.

References in corpus (13)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Reward Constrained Policy Optimization
• Maximum a Posteriori Policy Optimisation
• Equivalence Between Policy Gradients and Soft Q-Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Projection-Based Constrained Policy Optimization
• Supervised Policy Update for Deep Reinforcement Learning

Cited by in corpus (9)

• Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Policy Gradients for Probabilistic Constrained Reinforcement Learning
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• A Risk-Sensitive Approach to Policy Optimization
• Vision-driven UAV River Following: Benchmarking with Safe Reinforcement Learning
• Protective Policy Transfer