Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator

Kazuki Fujii, Kohei Watanabe, Rio Yokota

In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence…