SSDTrain: An Activation Offloading Framework to SSDs for Faster Large Language Model Training

Kun Wu, Jeongmin Brian Park, Xiaofan Zhang +5

The growth rate of the GPU memory capacity has not been able to keep up with that of the size of large language models (LLMs), hindering the model training process. In particular,…