DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism

Chenyu Jiang, Zhenkun Cai, Ye Tian +3

Context parallelism has emerged as a key technique to support long-context training, a growing trend in generative AI for modern large models. However, existing context parallel me…