2 papers
cs.DC2026
Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool
Yan Wang, Xiulong Yuan, Kaiming Yang +16
Long-context LLM training suffers from a load-balancing problem that sequence packing does not solve. Packing samples into fixed-token sequences balances memory and linear-cost ope…
cs.LG2026
SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
Hongtao Xu, Jianchao Tan, Yuxuan Hu +8
While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both \textit{1)} sequ…