Online Algorithms and Policies Using Adaptive and Machine Learning Approaches

Abstract

This paper considers the problem of real-time control and learning in dynamic systems subjected to parametric uncertainties. We propose a combination of a Reinforcement Learning (RL) based policy in the outer loop suitably chosen to ensure stability and optimality for the nominal dynamics, together with Adaptive Control (AC) in the inner loop so that in real-time AC contracts the closed-loop dynamics towards a stable trajectory traced out by RL. Two classes of nonlinear dynamic systems are considered, both of which are control-affine. The first class of dynamic systems utilizes equilibrium points %with expansion forms around these points and a Lyapunov approach while second class of nonlinear systems uses contraction theory. AC-RL controllers are proposed for both classes of systems and shown to lead to online policies that guarantee stability using a high-order tuner and accommodate parametric uncertainties and magnitude limits on the input. In addition to establishing a stability guarantee with real-time control, the AC-RL controller is also shown to lead to parameter learning with persistent excitation for the first class of systems. Numerical validations of all algorithms are carried out using a quadrotor landing task on a moving platform.

arXiv comment38 pages

References in corpus (17)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• A Variational Perspective on Accelerated Methods in Optimization
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Assessing Generalization in Deep Reinforcement Learning
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Direct Runge-Kutta Discretization Achieves Acceleration
• Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction
• Lyapunov Criterion for Stochastic Systems and Its Applications in Distributed Computation
• Towards Generalization and Simplicity in Continuous Control
• A Tour of Reinforcement Learning: The View from Continuous Control
• Accelerated Learning with Robustness to Adversarial Regressors
• Reinforcement Learning under Model Mismatch
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• A High-order Tuner for Accelerated Learning and Control