1 paper
Lu Zhao, Rong Shi, Shaoqing Zhang +14
The exponential growth in LLM scales, with parameters soaring from billions to trillions, has necessitated distributed pretraining across large clusters comprising thousands to ten…