Fast and Memory Efficient Offload Training Framework with Hybrid XPU Computation

Zhiyi Yao, Zuning Liang, Yuedong Xu +3

With the ever-growing size of deep learning models, GPU memory is prone to being insufficient during training. A prominent approach is ZeRO-Offload, which moves the optimizer state…