Taming LLMs by Scaling Learning Rates with Gradient Grouping

Siyuan Li, Juanxi Tian, Zedong Wang +4

Training large language models (LLMs) poses challenges due to their massive scale and heterogeneous architectures. While adaptive optimizers like AdamW help address gradient variat…