Pushing the boundaries of parallel Deep Learning -- A practical approach
arXiv:1806.09528
Abstract
This work aims to assess the state of the art of data parallel deep neural network training, trying to identify potential research tracks to be exploited for performance improvement. Beside, it presents a design for a practical C++ library dedicated at implementing and unifying the current state of the art methodologies for parallel training in a performance-conscious framework, allowing the user to explore novel strategies without departing significantly from its usual work-flow.
arXiv comment12 pages, 4 figures, 61 references
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Deep Learning in Neural Networks: An Overview
- Distilling the Knowledge in a Neural Network
- An overview of gradient descent optimization algorithms
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- cuDNN: Efficient Primitives for Deep Learning
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- The Loss Surfaces of Multilayer Networks
- Revisiting Distributed Synchronous SGD
- Don't Decay the Learning Rate, Increase the Batch Size
- Revisiting Small Batch Training for Deep Neural Networks
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
- Three Factors Influencing Minima in SGD
- Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
- Staleness-aware Async-SGD for Distributed Deep Learning
- Asynchronous Stochastic Gradient Descent with Delay Compensation
- GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
- Asynchronous Decentralized Parallel Stochastic Gradient Descent
- Accumulated Gradient Normalization