Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

Abstract

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

The arxiv version contains proofs of Lemma 3 and Lemma 5, which are missing in the published version

References in corpus (18)

• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback
• Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
• Coordinate Friendly Structures, Algorithms and Applications
• LQR through the Lens of First Order Methods: Discrete-time Case
• Distributed Non-Convex First-Order Optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms
• Angle-Displacement Rigidity Theory with Application to Distributed Network Localization
• Learning the Globally Optimal Distributed LQ Regulator
• Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training
• Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach
• On Topological and Metrical Properties of Stabilizing Feedback Gains: the MIMO Case
• Distributed Zero-Order Optimization under Adversarial Noise
• A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimization
• Privacy-Preserving Distributed Zeroth-Order Optimization
• Zeroth-Order Stochastic Block Coordinate Type Methods for Nonconvex Optimization