20 papers
Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Zhaoyi Li, Deyang Kong, Yuan Wei +13
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly unders…
Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation
Wei Zhang, Lin Tang, Ming Zhao +1
Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hu…
ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation
Chen Lin, Kedi Chen, Wei Zhang
On-policy distillation (OPD) improves LLM reasoning by training a student model on its own generated outputs, but standard OPD treats all student-generated outputs (SGOs) equally r…
From Recognition to Understanding: Unlocking Cognitive Time Series Reasoning with LLMs
Xin Qiu, Junlong Tong, Yao Zhang +3
Time series analysis has recently been coupled with Large Language Models (LLMs) to leverage their reasoning and world knowledge capabilities, yet gains remain limited. We attribut…
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Chao Han, Haozhe Hu, Fei Ma +2
Low-rank decomposition serves as a promising compression paradigm for large language models, however, rank allocation remains challenging: manual rules lack generalizability, and l…
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
Lin Tang, Wei Zhang, Jing Li +3
Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after b…