Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints

Abstract

This paper presents a constrained adaptive dynamic programming (CADP) algorithm to solve general nonlinear nonaffine optimal control problems with known dynamics. Unlike previous ADP algorithms, it can directly deal with problems with state constraints. Firstly, a constrained generalized policy iteration (CGPI) framework is developed to handle state constraints by transforming the traditional policy improvement process into a constrained policy optimization problem. Next, we propose an actor-critic variant of CGPI, called CADP, in which both policy and value functions are approximated by multi-layer neural networks to directly map the system states to control inputs and value function, respectively. CADP linearizes the constrained optimization problem locally into a quadratically constrained linear programming problem, and then obtains the optimal update of the policy network by solving its dual problem. A trust region constraint is added to prevent excessive policy update, thus ensuring linearization accuracy. We determine the feasibility of the policy optimization problem by calculating the minimum trust region boundary and update the policy using two recovery rules when infeasible. The vehicle control problem in the path-tracking task is used to demonstrate the effectiveness of this proposed method.

References in corpus (13)

• Continuous control with deep reinforcement learning
• An overview of gradient descent optimization algorithms
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• A Convergence Theory for Deep Learning via Over-Parameterization
• Reinforcement Learning with Deep Energy-Based Policies
• Learning Continuous Control Policies by Stochastic Value Gradients
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distributed Distributional Deterministic Policy Gradients
• Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Constrained Policy Optimization

Cited by in corpus (4)

• Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems
• Recurrent Model Predictive Control: Learning an Explicit Recurrent Controller for Nonlinear Systems
• Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
• Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning