How to Set the Learning Rate for Large-Scale Pre-training?

Yunhua Zhou, Shuhao Xing, Junhao Huang +2

Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. Given the stringent trade-off between training costs and mode…