Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach

Abstract

Model-free learning-based control methods have seen great success recently. However, such methods typically suffer from poor sample complexity and limited convergence guarantees. This is in sharp contrast to classical model-based control, which has a rich theory but typically requires strong modeling assumptions. In this paper, we combine the two approaches to achieve the best of both worlds. We consider a dynamical system with both linear and non-linear components and develop a novel approach to use the linear model to define a warm start for a model-free, policy gradient method. We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller.

References in corpus (22)

• End-to-End Training of Deep Visuomotor Policies
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• LQR through the Lens of First Order Methods: Discrete-time Case
• Residual Policy Learning
• Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification
• Learning-based Control of Unknown Linear Systems with Thompson Sampling
• Naive Exploration is Optimal for Online LQR
• Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
• Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• MBMF: Model-Based Priors for Model-Free Reinforcement Learning
• Learning Linear-Quadratic Regulators Efficiently with only $\sqrt{T}$ Regret
• Prediction and Control with Temporal Segment Models
• Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator
• Feedback Linearization for Unknown Systems via Reinforcement Learning
• Learning-based Model Predictive Control for Safe Exploration
• Uncertainty-aware Model-based Policy Optimization
• Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator

Cited by in corpus (4)

• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Stabilizing Dynamical Systems via Policy Gradient Methods
• Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient-Based Methods and Global Convergence
• Robust Reinforcement Learning: A Case Study in Linear Quadratic Regulation