Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

Egor Romanyukov, Timofey Novikov, Timur Shokarov +3

Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on models between 59M and 1B parame…