REG: A Regularization Optimizer for Robust Training Dynamics

Zehua Liu, Han Wu, Xiaojin Fu +4

Optimizers are crucial for the efficient training of Large Language Models (LLMs). While AdamW is the de facto standard, recent structure-aware optimizers like Muon have emerged, w…