Scaling Distributed Machine Learning with In-Network Aggregation

Abstract

Training machine learning models in parallel is an increasingly important workload. We accelerate distributed parallel training by designing a communication primitive that uses a programmable switch dataplane to execute a key step of the training process. Our approach, SwitchML, reduces the volume of exchanged data by aggregating the model updates from multiple workers in the network. We co-design the switch processing with the end-host protocols and ML frameworks to provide an efficient solution that speeds up training by up to 5.5$\times$ for a number of real-world benchmark models.

References in corpus (15)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• Exploring the Limits of Language Modeling
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• Revisiting Small Batch Training for Deep Neural Networks
• Distributed Coordinate Descent Method for Learning with Big Data
• Distributed Learning with Compressed Gradient Differences
• Training DNNs with Hybrid Block Floating Point
• NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration
• Revisiting Network Support for RDMA

Cited by in corpus (27)

• A Survey on Data Plane Programming with P4: Fundamentals, Advances, and Applied Research
• An Exhaustive Survey on P4 Programmable Data Plane Switches: Taxonomy, Applications, Challenges, and Future Trends
• Natural Compression for Distributed Deep Learning
• MIND: In-Network Memory Management for Disaggregated Data Centers
• Flare: Flexible In-Network Allreduce
• On Biased Compression for Distributed Learning
• Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms
• On the Utility of Gradient Compression in Distributed Training Systems
• A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent
• NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration
• Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol
• Domain-specific Communication Optimization for Distributed DNN Training
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
• DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
• Is Network the Bottleneck of Distributed Training?
• Toward an Abstract Model of Programmable Data Plane Devices
• Study on emerging applications on data plane and optimization possibilities
• LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
• Watching Smartly from the Bottom: Intrusion Detection revamped through Programmable Networks and Artificial Intelligence
• Running Neural Networks on the NIC
• Unleashing In-network Computing on Scientific Workloads
• Infinity: A Scalable Infrastructure for In-Network Applications
• A Roadmap for Enabling a Future-Proof In-Network Computing Data Plane Ecosystem
• SOAR: Minimizing Network Utilization with Bounded In-network Computing
• MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training
• RackSched: A Microsecond-Scale Scheduler for Rack-Scale Computers (Technical Report)
• Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters