Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP

Yilong Zhao, Xiaonan Nie, Kan Zhu +6

Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, existing designs fail to handle the large variation…