AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

Ling Chen, Houming Wu, Wenjie Yu

Pipeline parallelism is essential for large-scale model training, but existing asynchronous approaches often degrade convergence due to parameter mismatch between forward and backw…