Asynchronous Decentralized Learning of a Neural Network
arXiv:2004.05082
Abstract
In this work, we exploit an asynchronous computing framework namely ARock to learn a deep neural network called self-size estimating feedforward neural network (SSFN) in a decentralized scenario. Using this algorithm namely asynchronous decentralized SSFN (dSSFN), we provide the centralized equivalent solution under certain technical assumptions. Asynchronous dSSFN relaxes the communication bottleneck by allowing one node activation and one side communication, which reduces the communication overhead significantly, consequently increasing the learning speed. We compare asynchronous dSSFN with traditional synchronous dSSFN in the experimental results, which shows the competitive performance of asynchronous dSSFN, especially when the communication network is sparse.
References in corpus (8)
- Revisiting Distributed Synchronous SGD
- Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
- Training Neural Networks Without Gradients: A Scalable ADMM Approach
- Asynchronous Decentralized Parallel Stochastic Gradient Descent
- SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials
- A Data and Model-Parallel, Distributed and Scalable Framework for Training of Deep Networks in Apache Spark
- Distributed Averaging CNN-ELM for Big Data
- Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks