6 papers
From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
Hao Chen, Qi Zhang, Liyao Li +7
Adapting Large Language Models (LLMs) to specialized domains typically incurs high data and computational overhead. While prior efficiency efforts have largely treated data selecti…
Context Memorization for Efficient Long Context Generation
Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3
Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effe…
SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning
Run Zou, Jianhang Ding, Yifan Ding +3
Instruction tuning has optimized the specialized capabilities of large language models (LLMs), but it often requires extensive datasets and prolonged training times. The challenge…
Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority
Zhanming Shen, Zeyu Qin, Jiaqi Hu +7
The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is o…
Training-Trajectory-Aware Token Selection
Zhanming Shen, Jiaqi Hu, Zeyu Qin +7
Efficient distillation is a key pathway for converting expensive reasoning capability into deployable efficiency, yet in the frontier regime where the student already has strong re…
Merge-of-Thought Distillation
Zhanming Shen, Zeyu Qin, Zenan Huang +6
Efficient reasoning distillation for long chain-of-thought (CoT) models is increasingly constrained by the assumption of a single oracle teacher, despite the practical availability…