Modern Distributed Data-Parallel Large-Scale Pre-training Strategies For NLP models

Hao Bai

Distributed deep learning is becoming increasingly popular due to the expanding demand for computing resources for deep learning models with a larger amount of parameters. Differen…