MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training

Juntao Zhao, Qi Lu, Wei Jia +13

Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When pre…