HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware

Ran Yan, Youhe Jiang, Xiaonan Nie +3

Training large language models (LLMs) is a computationally intensive task, which is typically conducted in data centers with homogeneous high-performance GPUs. In this paper, we ex…