NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism

Yi'ou Wang, Xiaoyang Li, Yijie Zheng +2

Long-context LLM training on long-tailed corpora faces a central communication--balance tradeoff. Such sequence-length heterogeneity makes any single sequence-parallelism (SP) degr…