PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management

Jiarui Fang, Zilin Zhu, Shenggui Li +4

The pre-trained model (PTM) is revolutionizing Artificial Intelligence (AI) technology. However, the hardware requirement of PTM training is prohibitively high, making it a game fo…