29 citations · 30 across the 5 of their papers we have counts for
5 papers
REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
Haoran Que, Jiajun Shi, Ting Huang +7
As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follow…
When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning
Leheng Sheng, Yongtao Zhang, Wenchang Ma +6
While reasoning over long context is crucial for various real-world applications, it remains challenging for large language models (LLMs) as they suffer from performance degradatio…
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
Diandian Gu, Peng Sun, Qinghao Hu +11
Efficiently training LLMs with long sequences is important yet challenged by the massive computation and memory requirements. Sequence parallelism has been proposed to tackle these…
InternLM2 Technical Report
Zheng Cai, Maosong Cao, Haojiong Chen +97
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advan…
InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Qiaoling Chen, Diandian Gu, Guoteng Wang +8
Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are n…