Budget-aware Auto Optimizer Configurator

Kang Liu, Wei Peng, Jianchen Hu

Optimizer states occupy massive GPU memory in large-scale model training. However, gradients in different network blocks exhibit distinct behaviors, such as varying directional sta…