Lagrangian-based online safe reinforcement learning for state-constrained systems

Abstract

This paper proposes a safe reinforcement learning (RL) algorithm that approximately solves the state-constrained optimal control problem for continuous-time uncertain nonlinear systems. We formulate the safe RL problem as the minimization of a Lagrangian that includes the cost functional and a user-defined barrier Lyapunov function (BLF) encoding the state constraints. We show that the analytical solution obtained by the application of Karush-Kuhn-Tucker (KKT) conditions contains a state-dependent expression for the Lagrange multiplier, which is a function of uncertain terms in the system dynamics. We argue that a naive estimation of the Lagrange multiplier may lead to safety constraint violations. To obviate this challenge, we propose an Actor-Critic-Identifier-Lagrangian (ACIL) algorithm that learns optimal control policies from online data without compromising safety. We provide safety and boundedness guarantees with the proposed algorithm and compare its performance with existing offline/online RL methods via a simulation study.

References in corpus (9)

• Control Barrier Function Based Quadratic Programs for Safety Critical Systems
• Safe Reinforcement Learning Using Robust MPC
• Concurrent learning-based approximate optimal regulation
• Integral Concurrent Learning: Adaptive Control with Parameter Convergence without PE or State Derivatives
• Efficient model-based reinforcement learning for approximate online optimal
• Safe Exploration in Model-based Reinforcement Learning using Control Barrier Functions
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• Convergence Rate for a Radau hp Collocation Method Applied to Constrained Optimal Control
• Convergence rate for a Gauss collocation method applied to constrained optimal control