Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training

William Merrill, Shane Arora, Dirk Groeneveld +1

The right batch size is important when training language models at scale: a large batch size is necessary for fast training, but a batch size that is too large will harm token effi…