Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Abstract

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as \emph{integral} control; our terms introduce \emph{proportional} and \emph{derivative} control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

ICML 2020

References in corpus (10)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Continuous Deep Q-Learning with Model-based Acceleration
• Safe Exploration in Continuous Action Spaces
• Lyapunov-based Safe Policy Optimization for Continuous Control
• A General Analysis of the Convergence of ADMM
• Constrained Policy Optimization
• Projection-Based Constrained Policy Optimization
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• IPO: Interior-point Policy Optimization under Constraints

Cited by in corpus (9)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Conservative Safety Critics for Exploration
• Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
• FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimize
• Lyapunov-based uncertainty-aware safe reinforcement learning