One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

Di He, Songjun Tu, Keyu Wang +2

Learning rate configuration is a fundamental aspect of modern deep learning. The prevailing practice of applying a uniform learning rate across all layers overlooks the structural…