CAME: Confidence-guided Adaptive Memory Efficient Optimization

Yang Luo, Xiaozhe Ren, Zangwei Zheng +3

Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires m…