3 papers
cs.DC2024★ 1 cited
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
Diandian Gu, Peng Sun, Qinghao Hu +11
Efficiently training LLMs with long sequences is important yet challenged by the massive computation and memory requirements. Sequence parallelism has been proposed to tackle these…
cs.NE2024
RLEMMO: Evolutionary Multimodal Optimization Assisted By Deep Reinforcement Learning
Hongqiao Lian, Zeyuan Ma, Hongshu Guo +2
Solving multimodal optimization problems (MMOP) requires finding all optimal solutions, which is challenging in limited function evaluations. Although existing works strike the bal…
cs.DC2024
InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Qiaoling Chen, Diandian Gu, Guoteng Wang +8
Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are n…