BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

Bigyan Ghimire, Jon C. Calhoun

Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communica…