Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning

Lang Xu, Quentin Anthony, Jacob Hatef +4

Scaling up Large Language Model(LLM) training involves fitting a tremendous amount of training parameters across a limited number of workers. However, methods like ZeRO-3 that dras…