14 papers
Improving Cross-Format Robustness in Language Models with Multi-Format Training
June M. Liu, Shaomian Zheng, He Cao +3
Large language models often remain sensitive to answer format: a question solved correctly in one form may fail in another semantically equivalent form. To study this gap, we defin…
What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code
Yuze Zhao, Junpeng Fang, Lu Yu +6
Code has become a standard component of modern foundation language model (LM) training, yet its role beyond programming remains unclear. We revisit the claim that code improves rea…
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
Wen Lai, Yingli Shen, Dingnan Jin +4
Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only l…
On Representation Redundancy in Large-Scale Instruction Tuning Data Selection
Youwei Shu, Shaomian Zheng, Dingnan Jin +5
Data quality is a crucial factor in large language models training. While prior work has shown that models trained on smaller, high-quality datasets can outperform those trained on…
GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency
Changhao Wang, Jiaolong Yang, Xinhao Yao +7
The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple globa…
UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining
Changhao Wang, Yunfei Yu, Xinhao Yao +5
The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure…