2 citations · 2 across the 7 of their papers we have counts for
4 papers · 1 filter
Rethinking the Role of Temperature in Large Language Model Distillation
Hoang-Chau Luong, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons t…
Consistently Informative Soft-Label Temperature for Knowledge Distillation
Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1
Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a…
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL),…
Understanding SAM's Robustness to Noisy Labels through Gradient Down-weighting
Hoang-Chau Luong, Quang-Thuc Nguyen, Dat Ba Tran +1
Sharpness-Aware Minimization (SAM) was introduced to improve generalization by seeking flat minima, yet it also exhibits robustness to label noise, a phenomenon that remains only p…