dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training

Hanpeng Hu, Chenyu Jiang, Yuchen Zhong +5

Distributed training using multiple devices (e.g., GPUs) has been widely adopted for learning DNN models over large datasets. However, the performance of large-scale distributed tr…