ATOM: Asynchronous Training of Massive Models for Deep Learning in a Decentralized Environment

Xiaofeng Wu, Jia Rao, Wei Chen

The advent of the Transformer architecture has propelled the growth of natural language processing (NLP) models, leading to remarkable achievements in numerous NLP tasks. Yet, the…