Backward-Friendly Optimization: Training Large Language Models with Approximate Gradients under Memory Constraints

Jing Yang, Kaitong Cai, Yijia Fan +2

Full fine-tuning of Large Language Models (LLMs) is notoriously memory-intensive, primarily because conventional optimizers such as SGD or Adam assume access to exact gradients der…