Model-Free $\mu$ Synthesis via Adversarial Reinforcement Learning

Abstract

Motivated by the recent empirical success of policy-based reinforcement learning (RL), there has been a research trend studying the performance of policy-based RL methods on standard control benchmark problems. In this paper, we examine the effectiveness of policy-based RL methods on an important robust control problem, namely $\mu$ synthesis. We build a connection between robust adversarial RL and $\mu$ synthesis, and develop a model-free version of the well-known $DK$-iteration for solving state-feedback $\mu$ synthesis with static $D$-scaling. In the proposed algorithm, the $K$ step mimics the classical central path algorithm via incorporating a recently-developed double-loop adversarial RL method as a subroutine, and the $D$ step is based on model-free finite difference approximation. Extensive numerical study is also presented to demonstrate the utility of our proposed model-free algorithm. Our study sheds new light on the connections between adversarial RL and robust control.

arXiv commentAccepted to ACC 2022

References in corpus (11)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Robust Adversarial Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
• LQR through the Lens of First Order Methods: Discrete-time Case
• Learning the Globally Optimal Distributed LQ Regulator
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• From self-tuning regulators to reinforcement learning and back again