DHO${}_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM

Shunxian Gu, Chaoqun You, Bangbang Ren +3

Scaling deep neural network (DNN) training to more devices can reduce time-to-solution. However, it is impractical for users with limited computing resources. FOSI, as a hybrid ord…