An Efficient 2D Method for Training Super-Large Deep Learning Models

Qifan Xu, Shenggui Li, Chaoyu Gong +1

Huge neural network models have shown unprecedented performance in real-world applications. However, due to memory constraints, model parallelism must be utilized to host large mod…