19 citations · 37 across the 29 of their papers we have counts for
16 papers · 1 filter
Can Language Models Learn to Skip Steps?
Tengxiao Liu, Qipeng Guo, Xiangkun Hu +4
Trained on vast corpora of human language, language models demonstrate emergent human-like reasoning abilities. Yet they are still far from true intelligence, which opens up intrig…
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Jiangfei Duan, Shuo Zhang, Zerui Wang +13
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and sign…
ReAttention: Training-Free Infinite Context with Finite Attention Scope
Xiaoran Liu, Ruixiao Li, Qipeng Guo +7
The long-context capability of the Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length in length extrapolation remains a critic…
Case2Code: Scalable Synthetic Data for Code Generation
Yunfan Shao, Linyang Li, Yichuan Ma +11
Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs,…
In-Memory Learning: A Declarative Learning Framework for Large Language Models
Bo Wang, Tianxiang Sun, Hang Yan +3
The exploration of whether agents can align with their environment without relying on human-labeled data presents an intriguing research topic. Drawing inspiration from the alignme…
Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge
Jinlan Fu, Shenzhen Huangfu, Hang Yan +2
Large Language Models (LLMs) have recently showcased remarkable generalizability in various domains. Despite their extensive knowledge, LLMs still face challenges in efficiently ut…
LongWanjuan: Towards Systematic Measurement for Long Text Quality
Kai Lv, Xiaoran Liu, Qipeng Guo +4
The quality of training data are crucial for enhancing the long-text capabilities of foundation models. Despite existing efforts to refine data quality through heuristic rules and…
Turn Waste into Worth: Rectifying Top- Router of MoE
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei +7
Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top- routing mechanism su…
Code Needs Comments: Enhancing Code LLMs with Comment Augmentation
Demin Song, Honglin Guo, Yunhua Zhou +8
The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural…
Data-free Weight Compress and Denoise for Large Language Models
Runyu Peng, Yunhua Zhou, Qipeng Guo +4
Large Language Models (LLMs) are reshaping the research landscape in artificial intelligence, particularly as model parameters scale up significantly, unlocking remarkable capabili…
Balanced Data Sampling for Language Model Training with Clustering
Yunfan Shao, Linyang Li, Zhaoye Fei +3
Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sam…
Identifying Semantic Induction Heads to Understand In-Context Learning
Jie Ren, Qipeng Guo, Hang Yan +4
Although large language models (LLMs) have demonstrated remarkable performance, the lack of transparency in their inference logic raises concerns about their trustworthiness. To ga…
InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning
Huaiyuan Ying, Shuo Zhang, Linyang Li +19
The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which…
MouSi: Poly-Visual-Expert Vision-Language Models
Xiaoran Fan, Tao Ji, Changhao Jiang +21
Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issu…
F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods
Yu Sun, Keyu Chen, Shujie Wang +6
Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluat…
Unearthing Large Scale Domain-Specific Knowledge from Public Corpora
Zhaoye Fei, Yunfan Shao, Linyang Li +6
Large language models (LLMs) have demonstrated remarkable potential in various tasks, however, there remains a significant lack of open-source models and data for specific domains.…