OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning

Youhe Jiang, Fangcheng Fu, Xupeng Miao +2

Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model re…