Parle: parallelizing stochastic gradient descent
arXiv:1707.00424
Abstract
We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.
References in corpus (17)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Striving for Simplicity: The All Convolutional Net
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Densely Connected Convolutional Networks
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Wide Residual Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- One weird trick for parallelizing convolutional neural networks
- Deep Image: Scaling up Image Recognition
- Shake-Shake regularization
- Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- SparkNet: Training Deep Networks in Spark
- Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
- How to scale distributed deep learning?
- Learning may need only a few bits of synaptic precision
Cited by in corpus (8)
- Don't Use Large Mini-Batches, Use Local SGD
- A Field Guide to Federated Optimization
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
- Entropic gradient descent algorithms and wide flat minima
- Decentralized Bayesian Learning over Graphs
- Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
- The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication