5 papers · 1 filter
Rethinking the Role of Temperature in Large Language Model Distillation
Hoang-Chau Luong, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons t…
Consistently Informative Soft-Label Temperature for Knowledge Distillation
Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1
Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a…
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen
Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL),…
Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI
Christopher Burger, Charles Walter, Thai Le +1
Recent work has investigated the concept of adversarial attacks on explainable AI (XAI) in the NLP domain with a focus on examining the vulnerability of local surrogate methods suc…
DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs
Yingsong Luo, Ling Chen
Large language models (LLMs) excel in various tasks but face deployment challenges due to hardware constraints. We propose density-aware post-training weight-only quantization (DAQ…