DisDP: Disaggregating Compute, Network, and Storage for Model-Sharded Data-Parallel Training

Mo Sun, Zihan Yang, Changyue Liao +5

Model-sharded data parallelism (MSDP), e.g., ZeRO, evenly shards the model states across all GPUs, and thus has been widely adopted by LLM pre-training, such as Llama and DeepSeek,…