5 papers · 1 filter
Rethinking the Role of Temperature in Large Language Model Distillation
Hoang-Chau Luong, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons t…
Consistently Informative Soft-Label Temperature for Knowledge Distillation
Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1
Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a…
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL),…
Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals
Pengyue Yang, Jiawen Wen, Haolin Jin +3
Large language models (LLMs) are increasingly deployed in domains where errors carry high social, scientific, or safety costs. Yet standard confidence estimators, such as token lik…
Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models
Hoang-Chau Luong, Lingwei Chen
Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but it is notably ineffective at removing backdoor behaviors from poisoned p…