Gossip training for deep learning

Abstract

We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information between different threads inspired by gossip algorithms and showing good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. We compared our method to the recent EASGD in \cite{elastic} on CIFAR-10 show encouraging results.

References in corpus (1)

• The Loss Surfaces of Multilayer Networks

Cited by in corpus (14)

• Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets
• Decentralized Federated Learning: A Segmented Gossip Approach
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Collaborative Deep Learning in Fixed Topology Networks
• Decentralized federated learning of deep neural networks on non-iid data
• Communication-Efficient Edge AI: Algorithms and Systems
• EventGraD: Event-Triggered Communication in Parallel Machine Learning
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning
• Elastic Gossip: Distributing Neural Network Training Using Gossip-like Protocols
• A Framework for Energy and Carbon Footprint Analysis of Distributed and Federated Edge Learning
• Optimising cost vs accuracy of decentralised analytics in fog computing environments