Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games

Abstract

We propose projection-free sequential algorithms for linear-quadratic dynamics games. These policy gradient based algorithms are akin to Stackelberg leadership model and can be extended to model-free settings. We show that if the leader performs natural gradient descent/ascent, then the proposed algorithm has a global sublinear convergence to the Nash equilibrium. Moreover, if the leader adopts a quasi-Newton policy, the algorithm enjoys a global quadratic convergence. Along the way, we examine and clarify the intricacies of adopting sequential policy updates for LQ games, namely, issues pertaining to stabilization, indefinite cost structure, and circumventing projection steps.

References in corpus (3)

• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
• Actor-Critic Policy Optimization in Partially Observable Multiagent Environments

Cited by in corpus (10)

• Game-Theoretic Multiagent Reinforcement Learning
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs
• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• Independent Learning in Stochastic Games
• Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games
• The Confluence of Networks, Games and Learning
• Stability of Gradient Learning Dynamics in Continuous Games: Vector Action Spaces
• Approximate Midpoint Policy Iteration for Linear Quadratic Control