718 citations · 1.4k across the 11 of their papers we have counts for
4 papers · 1 filter
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Hanpeng Hu, Chenyu Jiang, Yuchen Zhong +5
Distributed training using multiple devices (e.g., GPUs) has been widely adopted for learning DNN models over large datasets. However, the performance of large-scale distributed tr…
Compressed Communication for Distributed Training: Adaptive Methods and System
Yuchen Zhong, Cong Xie, Shuai Zheng +1
Communication overhead severely hinders the scalability of distributed machine learning systems. Recently, there has been a growing interest in using gradient compression to reduce…
Is Network the Bottleneck of Distributed Training?
Zhen Zhang, Chaokun Chang, Haibin Lin +3
Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand wheth…
Just-in-Time Dynamic-Batching
Sheng Zha, Ziheng Jiang, Haibin Lin +1
Batching is an essential technique to improve computation efficiency in deep learning frameworks. While batch processing for models with static feed-forward computation graphs is s…