Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
arXiv:2004.14020
Abstract
The method of choice for parameter aggregation in Deep Neural Network (DNN) training, a network-intensive task, is shifting from the Parameter Server model to decentralized aggregation schemes (AllReduce) inspired by theoretical guarantees of better performance. However, current implementations of AllReduce overlook the interdependence of communication and computation, resulting in significant performance degradation. In this paper, we develop Caramel, a system that accelerates decentralized distributed deep learning through model-aware computation scheduling and communication optimizations for AllReduce. Caramel achieves this goal through (a) computation DAG scheduling that expands the feasible window of transfer for each parameter (transfer boundaries), and (b) network optimizations for smoothening of the load including adaptive batching and pipelining of parameter transfers. Caramel maintains the correctness of the dataflow model, is hardware-independent, and does not require any user-level or framework-level changes. We implement Caramel over TensorFlow and show that the iteration time of DNN training can be improved by up to 3.62x in a cloud environment.
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- One weird trick for parallelizing convolutional neural networks
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Rethinking the Inception Architecture for Computer Vision
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes