Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
arXiv:1805.07891 · doi:10.1145/3267809.3267840
Abstract
Distributed deep neural network (DDNN) training constitutes an increasingly important workload that frequently runs in the cloud. Larger DNN models and faster compute engines are shifting DDNN training bottlenecks from computation to communication. This paper characterizes DDNN training to precisely pinpoint these bottlenecks. We found that timely training requires high performance parameter servers (PSs) with optimized network stacks and gradient processing pipelines, as well as server and network hardware with balanced computation and communication resources. We therefore propose PHub, a high performance multi-tenant, rack-scale PS design. PHub co-designs the PS software and hardware to accelerate rack-level and hierarchical cross-rack parameter exchange, with an API compatible with many DDNN training frameworks. PHub provides a performance improvement of up to 2.7x compared to state-of-the-art distributed training techniques for cloud-based ImageNet workloads, with 25% better throughput per dollar.
References in corpus (12)
- TensorFlow: A system for large-scale machine learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Revisiting Distributed Synchronous SGD
- Training Deep Nets with Sublinear Memory Cost
- Horovod: fast and easy distributed deep learning in TensorFlow
- Benchmarking State-of-the-Art Deep Learning Software Tools
- TBD: Benchmarking and Analyzing Deep Neural Network Training
- Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
- DimmWitted: A Study of Main-Memory Statistical Analytics
- DeepConfig: Automating Data Center Network Topologies Management with Machine Learning