On the Performance and Memory Footprint of Distributed Training: An Empirical Study on Transformers

Zhengxian Lu, Fangyu Wang, Zhiwei Xu +2

Transformer models have emerged as potent solutions to a wide array of multidisciplinary challenges. The deployment of Transformer architectures is significantly hindered by their…