The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication

Abstract

The paper considers independent reinforcement learning (IRL) for multi-agent collaborative decision-making in the paradigm of federated learning (FL). However, FL generates excessive communication overheads between agents and a remote central server, especially when it involves a large number of agents or iterations. Besides, due to the heterogeneity of independent learning environments, multiple agents may undergo asynchronous Markov decision processes (MDPs), which will affect the training samples and the model's convergence performance. On top of the variation-aware periodic averaging (VPA) method and the policy-based deep reinforcement learning (DRL) algorithm (i.e., proximal policy optimization (PPO)), this paper proposes two advanced optimization schemes orienting to stochastic gradient descent (SGD): 1) A decay-based scheme gradually decays the weights of a model's local gradients with the progress of successive local updates, and 2) By representing the agents as a graph, a consensus-based scheme studies the impact of exchanging a model's local gradients among nearby agents from an algebraic connectivity perspective. This paper also provides novel convergence guarantees for both developed schemes, and demonstrates their superior effectiveness and efficiency in improving the system's utility value through theoretical analyses and simulation results.

References in corpus (18)

• A Brief Survey of Deep Reinforcement Learning
• Federated Learning: Strategies for Improving Communication Efficiency
• Trust Region Policy Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Optimal Distributed Online Prediction using Mini-Batches
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Local SGD Converges Fast and Communicates Little
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradient
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Parle: parallelizing stochastic gradient descent
• Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms