3 papers
cs.LG2026
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
Xin Yu, Liuchen Liao, Yiwen Zhang +3
On-policy distillation is an efficient alternative to reinforcement learning, offering dense token-level training signals. However, its reliance on a stronger external teacher has…
cs.AI2025
Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs
Yao Fu, Xianxuan Long, Runchao Li +5
Quantization enables efficient deployment of large language models (LLMs) in resource-constrained environments by significantly reducing memory and computation costs. While quantiz…
cs.CL2024
Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
Yao Fu, Yin Yu, Xiaotian Han +4
Knowledge distillation (KD) has become a widely adopted approach for compressing large language models (LLMs) to reduce computational costs and memory footprints. However, the avai…