SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

Qiulin Shang, Binyu Wang, Yongqi Qiao +3

Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Co…