3 papers
cs.DC2026
Accelerating MoE Training through Expert Placement and Replica Coordination
Zhiqiang Tan, Maoxin Wang, Wenxiang Lin +4
Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token r…
cs.LG2023
Eva: A General Vectorized Approximation Framework for Second-order Optimization
Lin Zhang, Shaohuai Shi, Bo Li
Second-order optimization algorithms exhibit excellent convergence properties for training deep learning models, but often incur significant computation and memory overheads. This…
cs.LG2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning
Lin Zhang, Shaohuai Shi, Wei Wang +1
The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural networ…