Distributed Sign Momentum with Local Steps for Training Transformers

Shuhua Yu, Ding Zhou, Cong Xie +4

Pre-training Transformer models is resource-intensive, and recent studies have shown that sign momentum is an efficient technique for training large-scale deep learning models, par…