Rethinking Memory and Communication Cost for Efficient Large Language Model Training

Chan Wu, Hanxiao Zhang, Lin Ju +8

Recently, various distributed strategies for large language model training have been proposed. However, these methods provided limited solutions for the trade-off between memory co…