1.5k citations · 1.8k across the 22 of their papers we have counts for
23 papers
FARS: A Fully Automated Research System Deployed at Scale
Qiong Tang, Tianxiang Sun, Xiangkun Hu +54
Recent automated research systems show that language-model agents can generate hypotheses, run experiments, and write complete manuscripts, but most evidence still comes from selec…
Position Bias Correction is Insufficient for One-Pass Attention Sorting
Qiong Tang, Xiangkun Hu, Xiangyang Liu +2
Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents…
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Qiong Tang, Xiangkun Hu, Xiangyang Liu +2
Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of \emph{…
Output-Space Allocation Costs for Calibration-Guided LLM Compression: An Empirical Study
Qiong Tang, Xiangkun Hu, Xiangyang Liu +2
Training-free compression methods for large language models (LLMs) often use calibration data to guide compression decisions. ROCKET, a recent method combining sparse-dictionary fa…
UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance
Yichuan Ma, Yunfan Shao, Peiji Li +5
Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge. Current approaches for obtaining high-qualit…
FastMCTS: A Simple Sampling Strategy for Data Synthesis
Peiji Li, Kai Lv, Yunfan Shao +5
Synthetic high-quality multi-step reasoning data can significantly enhance the performance of large language models on various tasks. However, most existing methods rely on rejecti…
GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?
Zhikai Lei, Tianyi Liang, Hanglei Hu +8
Large Language Models (LLMs) are commonly evaluated using human-crafted benchmarks, under the premise that higher scores implicitly reflect stronger human-like performance. However…
Case2Code: Scalable Synthetic Data for Code Generation
Yunfan Shao, Linyang Li, Yichuan Ma +11
Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs,…
Unified Active Retrieval for Retrieval Augmented Generation
Qinyuan Cheng, Xiaonan Li, Shimin Li +7
In Retrieval-Augmented Generation (RAG), retrieval is not always helpful and applying it to every instruction is sub-optimal. Therefore, determining whether to retrieve is crucial…
InternLM2 Technical Report
Zheng Cai, Maosong Cao, Haojiong Chen +97
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advan…
Balanced Data Sampling for Language Model Training with Clustering
Yunfan Shao, Linyang Li, Zhaoye Fei +3
Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sam…
InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning
Huaiyuan Ying, Shuo Zhang, Linyang Li +19
The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which…
Unearthing Large Scale Domain-Specific Knowledge from Public Corpora
Zhaoye Fei, Yunfan Shao, Linyang Li +6
Large language models (LLMs) have demonstrated remarkable potential in various tasks, however, there remains a significant lack of open-source models and data for specific domains.…
PerturbScore: Connecting Discrete and Continuous Perturbations in NLP
Linyang Li, Ke Ren, Yunfan Shao +2
With the rapid development of neural network applications in NLP, model robustness problem is gaining more attention. Different from computer vision, the discrete nature of texts m…
Character-LLM: A Trainable Agent for Role-Playing
Yunfan Shao, Linyang Li, Junqi Dai +1
Large language models (LLMs) can be used to serve as agents to simulate human behaviors, given the powerful ability to understand human instructions and provide high-quality genera…
Black-Box Tuning for Language-Model-as-a-Service
Tianxiang Sun, Yunfan Shao, Hong Qian +2
Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service. It allows users to design task-specific prompts to query the PTMs through some b…
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
Yunfan Shao, Zhichao Geng, Yitao Liu +6
In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTM…
Accelerating BERT Inference for Sequence Labeling via Early-Exit
Xiaonan Li, Yunfan Shao, Tianxiang Sun +3
Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the…
Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces
Linyang Li, Yunfan Shao, Demin Song +2
Adversarial attacks in texts are mostly substitution-based methods that replace words or characters in the original texts to achieve success attacks. Recent methods use pre-trained…
CoLAKE: Contextualized Language and Knowledge Embedding
Tianxiang Sun, Yunfan Shao, Xipeng Qiu +4
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained…
Pre-trained Models for Natural Language Processing: A Survey
Xipeng Qiu, Tianxiang Sun, Yige Xu +3
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP.…
Learning Sparse Sharing Architectures for Multiple Tasks
Tianxiang Sun, Yunfan Shao, Xiaonan Li +4
Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanis…
Star-Transformer
Qipeng Guo, Xipeng Qiu, Pengfei Liu +3
Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In…