AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control

Quang-Hung Bui, Anh Son Ta

Training Large Language Models (LLMs) is highly memory-intensive due to optimizer state overhead. The FRUGAL framework mitigates this with gradient splitting, but its static hyperp…