6 papers · 1 filter
REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
Haoran Que, Jiajun Shi, Ting Huang +7
As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follow…
Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Zhaoyi Li, Deyang Kong, Yuan Wei +13
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly unders…
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Mengyuan Sun, Yu Li, Zhuohao Yu +2
Rubric-based evaluation is a promising paradigm for judging large language model (LLM) outputs, yet self-generated rubrics lag human-annotated criteria on hard instances. We argue…
Retrieval as Generation: A Unified Framework with Self-Triggered Information Planning
Bo Li, Mingda Wang, Gexiang Fang +2
We revisit retrieval-augmented generation (RAG) by embedding retrieval control directly into generation. Instead of treating retrieval as an external intervention, we express retri…
Instruction Data Selection via Answer Divergence
Bo Li, Mingda Wang, Shikun Zhang +1
Instruction tuning relies on large instruction-response corpora whose quality and composition strongly affect downstream performance. We propose Answer Divergence-Guided Selection…
Scaling Latent Reasoning via Looped Language Models
Rui-Jie Zhu, Zixuan Wang, Kai Hua +30
Modern LLMs are trained to "think" primarily via explicit text generation, such as chain-of-thought (CoT), which defers reasoning to post-training and under-leverages pre-training…