Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation

Tianyuan Wu, Lunxi Cao, Hanfeng Lu +8

Training large Deep Neural Network (DNN) models at scale often encounters straggler issues, mostly in communications due to network congestion, RNIC/switch defects, or topological…