Maximizing Parallelism in Distributed Training for Huge Neural Networks

Zhengda Bian, Qifan Xu, Boxiang Wang +1

The recent Natural Language Processing techniques have been refreshing the state-of-the-art performance at an incredible speed. Training huge language models is therefore an impera…