Distributed Deep Learning with Event-Triggered Communication
arXiv:1909.05020
Abstract
We develop a Distributed Event-Triggered Stochastic GRAdient Descent (DETSGRAD) algorithm for solving non-convex optimization problems typically encountered in distributed deep learning. We propose a novel communication triggering mechanism that would allow the networked agents to update their model parameters aperiodically and provide sufficient conditions on the algorithm step-sizes that guarantee the asymptotic mean-square convergence. The algorithm is applied to a distributed supervised-learning problem, in which a set of networked agents collaboratively train their individual neural networks to recognize handwritten digits in images, while aperiodically sharing the model parameters with their one-hop neighbors. Results indicate that all agents report similar performance that is also comparable to the performance of a centrally trained neural network, while the event-triggered communication provides significant reduction in inter-agent communication. Results also show that the proposed algorithm allows the individual agents to recognize the digits even though the training data corresponding to all the digits are not locally available to each agent.
arXiv commentarXiv admin note: substantial text overlap with arXiv:1908.06693
References in corpus (16)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Parallel and Distributed Methods for Nonconvex Optimization--Part II: Applications
- D: Decentralized Training over Decentralized Data
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Stochastic Gradient Push for Distributed Deep Learning
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- First-order Methods Almost Always Avoid Saddle Points
- On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
- MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
- NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization
- Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization
- Sharp Analysis for Nonconvex SGD Escaping from Saddle Points
- Distributed Stochastic Gradient Tracking Methods