TAGC: Optimizing Gradient Communication in Distributed Transformer Training

Igor Polyakov, Alexey Dukhanov, Egor Spirin

The increasing complexity of large language models (LLMs) necessitates efficient training strategies to mitigate the high computational costs associated with distributed training.…