4 papers · 1 filter
Training-Trajectory-Aware Token Selection
Zhanming Shen, Jiaqi Hu, Zeyu Qin +7
Efficient distillation is a key pathway for converting expensive reasoning capability into deployable efficiency, yet in the frontier regime where the student already has strong re…
Context Memorization for Efficient Long Context Generation
Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3
Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effe…
SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning
Run Zou, Jianhang Ding, Yifan Ding +3
Instruction tuning has optimized the specialized capabilities of large language models (LLMs), but it often requires extensive datasets and prolonged training times. The challenge…
Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority
Zhanming Shen, Zeyu Qin, Jiaqi Hu +7
The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is o…