AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training

Fu-Ming Guo, Yingfang Fan

Adaptive optimizers with decoupled weight decay, such as AdamW, are the de facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of th…