FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training

Zheng Wang, Eric Liu, Linan Jiang +5

Context parallelism (CP) is essential for training large-scale, long-context language models, as it partitions sequences to reduce memory overhead. However, existing CP methods suf…