FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

Zhenheng Tang, Xueze Kang, Yiming Yin +11

To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system desig…