MoLS: SNR-Guided Module-Wise Learning-Rate Calibration for Large Language Models

Ziqing Wen, Zhouyang Liu, Jiahuan Wang +5

Large language models (LLMs) contain functionally distinct modules with heterogeneous gradient statistics. Although Adam(W) adapts updates at the parameter level, selecting relativ…