Beyond First-Order: Training LLMs with Stochastic Conjugate Subgradients and AdamW

Di Zhang, Yihang Zhang

Stochastic gradient-based descent (SGD), have long been central to training large language models (LLMs). However, their effectiveness is increasingly being questioned, particularl…