AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

Huishuai Zhang, Bohan Wang, Luoxin Chen

We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training. By leveraging a novel denominator, i.e., the root of we…