45 citations · 75 across the 10 of their papers we have counts for
Showing 2025Show all
2 papers · 1 filter
cs.LG2025
AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
Fu-Ming Guo, Yingfang Fan
Adaptive optimizers with decoupled weight decay, such as AdamW, are the de facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of th…
cs.CL2025
dots.llm1 Technical Report
Bi Huo, Bin Tu, Cheng Qin +24
Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this…