Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training

Yishun Lu, Junhao Zhang, Zeyu Yang +1

Second-order methods offer an attractive path toward more sample-efficient LLM training, but their practical use is often blocked by the systems cost of maintaining and updating la…