Scaling Law for Language Models Training Considering Batch Size

Xian Shuai, Yiding Wang, Yimeng Wu +2

Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress. In this paper, we empirically investig…