Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization

Huiping Zhuang, Zhiping Lin, Kar-Ann Toh

Decoupled learning is a branch of model parallelism which parallelizes the training of a network by splitting it depth-wise into multiple modules. Techniques from decoupled learnin…